Issue with exporting multiple renders

I have a weird issue where when exporting multiple videos, the first video is always 4-7x larger in file size compared to the rest of the videos. All videos are rendered with the same setting, at the same time, to the same location, and are all roughly the same duration.
I appreciate any insight