eliminating the frame gap in time lapse video?

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

eliminating the frame gap in time lapse video?

PostSat Mar 03, 2018 12:10 am

]Greetings,

Please forgive my ignorance. I’m just starting out as a developer and I’m experiencing issues that I don’t know how to solve.

I’d like to create a light accumulation app that behaves like traditional time lapse photography., but I’m experiencing gaps in the frames.
BMDcomp.jpg
BMDcomp.jpg (91.78 KiB) Viewed 4333 times


While researching this I discovered that it may be related to the space that holds VANC or ancillary data.

Can someone confirm or refute this? Also, is it possible to achieve this by modifying the SDK code? Is it as simple as commenting out any reference to including ancillary data, or is it more involved? Any code or information appreciated. This is for a video only project so I’m not concerned about including audio in the stream.

Thanks for your time.
vidTimSpec.jpg
vidTimSpec.jpg (79.18 KiB) Viewed 4333 times

timeline0.jpg
timeline0.jpg (19.78 KiB) Viewed 4333 times
Offline

Andy Coulthurst

  • Posts: 273
  • Joined: Thu Apr 21, 2016 12:04 pm

Re: eliminating the frame gap in time lapse video?

PostSat Mar 03, 2018 8:34 pm

I think we need more detail about the imaging system ( camera ) and how you are triggering a frame capture.

Are you waiting for a period of time determined by the computer and then capturing a frame ? If you are then I suspect you really need to continiously capture frames , and then drop every N frames to get your result.
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostSun Mar 04, 2018 11:12 pm

Thanks Andy,
I’m running an HDMI feed out from a DSLR through the UltraStudio Mini Recorder to the computer via Thunderbolt. From there I’m using BlackSyphon to pipe the video into my main app that processes the loop.

I believe the loop process is pretty solid. I’ve run a GL shape through it and got the results I expected.
Screen Shot 2018-02-15 at 2.21.28 PM.png
Screen Shot 2018-02-15 at 2.21.28 PM.png (10.24 KiB) Viewed 4281 times

As for the time period to capturing a frame, I’m relying on the stock configuration of all the equipment in the chain. The HDMI sends interlaced 59.94, although I have experimented with Magic Lantern to manipulate the frame rate of the captured video. But even at 1 frame per second capture there is a break in the image:
Screen Shot 2018-03-04 at 5.13.07 PM.png
Screen Shot 2018-03-04 at 5.13.07 PM.png (177.53 KiB) Viewed 4281 times


Based on the graph I posted earlier, lead me to believe that the general time interval from line blanking was the issue. This is only a theory and I was hoping someone with more experience can confirm or refute this. As far as I know I am continuously capturing frames. I’m not sure that dropping frames is the solution. In a previous example I started drawing the B from the lower left corner, and by the second loop through there is already a gap. I think by dropping frames the gap would be more distinct, no?

As I was browsing through the SDK I found a few related features:
page 227:

BMDDeckLinkSupportsClockTimingAdjustment - True if this device supports clock timing adjustment (see bmdDeckLinkConfigClockTimingAdjustment).

pg 231
bmdDeckLinkConfigClockTimingAdjustment - Clock frequency adjustment for fine output control. The acceptable range is from -127 to 127 PPM (Parts Per Million).

pg 228
bmdDeckLinkConfigReferenceInputTimingOffset - Adjust genlock timing pixel offset. If the device supports wide genlock offset adjustment (see BMDDeckLinkSupportsFullFrameReferenceInput TimingOffset attribute) then the supported range is between +/- half the count of total pixels in the video frame. Otherwise the supported range is +/- 511.

pg 229
bmdDeckLinkConfigVANCSourceLine1Mapping - The configuration of up to three lines of VANC to be transferred to or from the active picture on capture or output. The acceptable range is between 0 and 30. A value of 0 will disable the capture of that line.

pg 231
bmdDeckLinkConfigClockTimingAdjustment - Clock frequency adjustment for fine output control. The acceptable range is from -127 to 127 PPM (Parts Per Million).

My challenge is that I’m such a novice at programming that it will probably take a few months (years?) to wrap my head around all this stuff… and then… there’s no guarantee that I would be successful. Additionally, I need to find out if the Mini Recorder supports these timing adjustments, or do I need to purchase a different device. Thanks for the feedback!
Offline

Andy Coulthurst

  • Posts: 273
  • Joined: Thu Apr 21, 2016 12:04 pm

Re: eliminating the frame gap in time lapse video?

PostMon Mar 05, 2018 7:10 pm

Just a quick few points ( sorry I have to rush ).
The video from your camera will always be at the output rate e.g. 59.94 even if you set the camera to capture at 1s intervals. The video data in the camera frame buffer will be updated once per second , but you will get 59.94 frames per second into the capture card over the hdmi.
Horizontal blanking/data is nothing to do with this. I think you are experiencing non synchronised events between changing of image ( camera taking a picture ) and you issuing the capture of the frame.
How do you ensure that the taking of a picture is synchronised with the capture of the video frame ?

Have you tried capturing live video from the camera to a video file ? And processing this ? What happens ?
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostWed Mar 07, 2018 1:43 am

Thanks for taking the time to educate me. I know this is a fairly esoteric process and I appreciate your expertise and feedback.


I have captured live video to a file and post-processed it. I get the same results as if I was processing it live. Another experiment I tried was exporting the unprocessed video as an image sequence, then stacking the images in photoshop layers (using the appropriate blending mode to see all the frames). In this case I got the same result - each frame was ‘off’ by about 5 pixels (1920x1080).

" I think you are experiencing non synchronised events between changing of image ( camera taking a picture ) and you issuing the capture of the frame.”

Just to be clear, the images coming from the camera are generated from the LiveView function of the camera. As for synchronization I’m relying on the stock settings of both the camera and the Mini Recorder. This is why I’m looking at the BMD SDK to see if I can manipulate the software to eliminate this gap. Also, to be clear all of the equipment involved is working as designed. Traditional video capture and playback work as expected. It’s only when I capture/loop single frames I experience this gap.

If I was to play with this synchronized playback/capture what part of the code should I be manipulating?
Offline

Andy Coulthurst

  • Posts: 273
  • Joined: Thu Apr 21, 2016 12:04 pm

Re: eliminating the frame gap in time lapse video?

PostWed Mar 07, 2018 6:48 am

I can’t be sure but it feels like something (presumably the camera ) is doing a frame rate conversion and dropping frames.

When you recorded video for post processing I assume you used the blackmagic utility rather than using the SDK.
If you monitor live view via hdmi ( use a monitor directly ) and perform fast pans with the camera - does the result look smooth or jerky ?
You’ve really got to eliminate some of the complexity to work this one out - sorry I cannot help any more.

If you are relying on the camera / mini recorder for frame interval synchronisation you could be a frame out - which is why I suggested you should capture continiously and drop every N frames to achieve interval recording - that way the distance in frames between captures would be consistent and not have the time gaps you see in your original images.
Last edited by Andy Coulthurst on Wed Mar 07, 2018 6:59 am, edited 1 time in total.
Offline

Andy Coulthurst

  • Posts: 273
  • Joined: Thu Apr 21, 2016 12:04 pm

Re: eliminating the frame gap in time lapse video?

PostWed Mar 07, 2018 6:57 am

I seem to recall that sdk capture requires capturing/buffering several frames before you get frames out - but its been years since I’ve written code for the capture sdk. So if you start and stop capture you may not have a consistent interval in frames between images.
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostWed Mar 07, 2018 6:38 pm

Thanks Andy. Yes, I have a lot of variables to work out. Appreciate the feedback!
Offline

Dave Johnstone

  • Posts: 223
  • Joined: Mon Dec 08, 2014 11:12 am
  • Location: Adelaide

Re: eliminating the frame gap in time lapse video?

PostWed Mar 07, 2018 9:36 pm

This could be a mismatch between the camera shooting rate and output rate and/or shutter speed. You could try a low shutter speed.

Regards,

Dave.
Freelance Television Broadcast Engineer
and Software Programmer.
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostThu Mar 08, 2018 4:07 am

Hi Dave,

Thanks for the input.

I’m mostly writing this so I can collect my thoughts, but feel free to chime in if you have something to add or refute. I don’t know all the technical terms for gaps and I’m quite sure there’s a bunch of unwanted ways to get gaps, but here’s what I found:

The shutter speed (anywhere from 1/8 to 1/4000) only affects the overall exposure, not the frame duration.

Asynchronous Gaps
This occurs when the looping capture/render rate is slower than the frame rate. For 29.97fps (33.36ms) The buffer capture needs to render every 14ms.. For 23.976fps (41.708ms) I can slow down the buffer capture to 20ms.

Not sure what that all means, but something logarithmic(?) is going on depending on the frame rate of the camera.

Micro Gaps
Beyond that there are ‘micro gaps’(?) Very uniform gaps whose duration is also based on the frame rate. For 29.97fps they’re about 5 pixels late. For a 1 second frame the pixels are about 30 pixels late. (relative to a 1920x1080 frame)

The Holy Grail I’m trying to achieve is the far right image.

Test Images
A 2ft. LED strip rotated around its center for approx. 1 second. (Yes, there’s an aspect ratio tweak I need to make.)
frameGaps03.jpg
frameGaps03.jpg (171.99 KiB) Viewed 4219 times

In trying to figure this out I can’t help going back to the graph that illustrates how each frame contains non image data. Measured in microSeconds. I’m hoping that broadcast control can give me sub-frame(?) access upon OUTPUT(?) by sliding each newFrame x amount depending on the frame rate. And by ‘sliding’ it I mean asking the program:

Hey DeckLink,

Capture what you want, but please output a video stream that has truncated the non-image area of each frame and stretch the visual picture area if necessary.

Thank you.

vidTimSpec.jpg
vidTimSpec.jpg (79.18 KiB) Viewed 4219 times
Offline

Baldur Gislason

  • Posts: 13
  • Joined: Sat Jan 06, 2018 7:55 pm

Re: eliminating the frame gap in time lapse video?

PostThu Mar 08, 2018 1:01 pm

What it really boils down to is your shutter must remain open for the entire duration of the frame and there must not be any gaps between frames. Any light that isn't caught in a frame is going to create those artifacts you showed.
Baldur Gislason
Offline
User avatar

Fred Rodrigues

  • Posts: 364
  • Joined: Mon Nov 05, 2012 9:00 am
  • Location: Amsterdam

Re: eliminating the frame gap in time lapse video?

PostThu Mar 08, 2018 10:40 pm

The shutter speed does matter as it is the part that is capturing time. If the shutter speed is fast, say 1/1000 then each frame misses a lot of information. If the shutter is the same duration as a frame (I think this i refered to as 180 degrees in film world) and you use reference to sync the camera video output and the capture input you will still get the gaps. The shutter data has to be cleared and then new information fed into the video feed. Although it is very small, this does take time. The solution would be to use a shutter speed that is longer than a frame and have the camera deal wirth clearing its buffer. This is available on many cameras, sometimes called SLS, DSS slow shutter, or sometimes you see the ability to set the shutter speed as 1/6 of a second. When you see this image on the camera it looks like motion blur, but you will get the video feed you want with no gaps.

Lets say you had a shutter duratioin of 2 frames and the camera had a ping pong buffer to deal with this, every "frame" of data that came through the video output would either contain all of its data and all of the data from the previous frame, or more likely (as sychronising with reference does not sychronise shutters, just video pulses) some of the data collected in the previously sent frame, all of the data in the current frame, and some of the data in what would be the next frame when it is complete.

Go with a long exposure and the overlap should assure you that you dont get gapss.

At the moment, as the shutter opening and closing is a separate proces to the frame being built and sent to the cameras SDI out the transition will never be exacty on time.
http://www.fredrodrigues.net/
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostFri Mar 09, 2018 2:13 pm

Thanks Baldur and Fred.

What I’m finding is that even if the shutter is open for the entire duration of the frame, I’m still dealing with the gap between frames. As Andy pointed out “even if you set the camera to capture at 1s intervals… the video data in the camera buffer will be updated once per second, but you will get x frames per second into the capture card..”

Here’s my test.
29.97fps @ 1/15 and at 1 second. The 1 second test was 4 stops brighter so I stopped down the aperture to compensate. The gap lines shown are from the 29.97fps cycle, regardless of the shutter speed.
sSpeed.jpg
sSpeed.jpg (35.45 KiB) Viewed 4125 times


I’m curious about the ping pong buffer idea.
Let’s say I set the camera to capture at 30fps. So every 33.33ms it sends a frame. If I set my capture to ping pong between two 16.66ms buffers, one buffer may have most of the light and the other buffer may have only part of the light and the remaining inter frame gap. There doesn’t seem to be much wiggle room for getting all of the light in one buffer. So.. let’s break this up into 34 buffers to be sure I have all of the image area.

According to the broadcast specs, the visual picture area is about 81.25% of the entire frame so there’s really only 24 buffers that contain light. Regardless, I should be able to blend these 34 buffers into one frame.

But this won’t work unless I blend two sets of 34 buffers... offset by 12us (microSeconds) because the first set needs the full duration to ensure capture…. Aaargh!

It seems like some of this is built into the DECKLINK SDK:

pg 228
bmdDeckLinkConfigReferenceInputTimingOffset - Adjust genlock timing pixel offset. If the device supports wide genlock offset adjustment (see BMDDeckLinkSupportsFullFrameReferenceInput TimingOffset attribute) then the supported range is between +/- half the count of total pixels in the video frame. Otherwise the supported range is +/- 511.

BMDDeckLinkSupportsClockTimingAdjustment - True if this device supports clock timing adjustment (see bmdDeckLinkConfigClockTimingAdjustment).

pg 231
bmdDeckLinkConfigClockTimingAdjustment - Clock frequency adjustment for fine output control. The acceptable range is from -127 to 127 PPM (Parts Per Million).

Does anyone have any experience implementing these settings? Yes, I’ll check with support.
Offline

stephen.schweitzer

  • Posts: 8
  • Joined: Mon Feb 20, 2017 10:46 pm

Re: eliminating the frame gap in time lapse video?

PostTue Mar 13, 2018 9:55 pm

Well this is interesting…

I wanted to get a sense of how long the gap is relative to the image area. I used a laser pen to shine light along the horizontal and vertical frame to see if there was a difference in the gap. There seemed to be no discernible difference.

Then I tried changing the aperture - wow!

With a Canon electric lens there was a huge difference in the gap length. At f.2.8 the gap is minimal, but at f.22 the gap is almost half the image area.
canon.jpg
canon.jpg (20.04 KiB) Viewed 3926 times


Then I tried a few non-electric lenses and changing the aperture seemed to have effect on frame gap (other than a difference in exposure).

nonElec.jpg
nonElec.jpg (19.9 KiB) Viewed 3926 times


Interesting! In addition to the frame rate, ISO and shutter speed, It seems the Canon electronics also control the exposure by manipulating the duration of light based on the aperture setting. A wider aperture includes more image area per frame, where a smaller aperture truncates(?) the amount of light... leaving a lengthier gap per frame.

Not sure what this all means, but just another variable to contend with.

All of this was tested on a Canon 7D HDMI out through the Mini Recorder. Captured by Black Syphon and piped through my looping software.

Well… back to Objective C and C++ tutorials.

Return to Software Developers

Who is online

Users browsing this forum: No registered users and 15 guests