bmdModeNTSC works well, but others not

Hi, We're developing the preview program plays sdi signals by using .Net and WPF on Windows 10.
The signals from the video server for play-out.
I'm using IDeckLinkInputCallback::VideoInputFrameArrived() for gettting video and audio data like the CapturePreviewCSharp SDK sample.

Our problem was choppy audio sound.
I just pass audio from BMD DeckLink to the speaker(exactly monitor with HDMI) directly by setting 'Listen to this device' on Sound of Control Panel, so there's no codes in the app for audio processing.
Like the sample I run IDeckLinkInput::EnableVideoInput() with BMDVideoInputFlags.bmdVideoInputEnableFormatDetection flag for start capturing.
With that flag, the problem has happened.
It's the same with the CapturePreviewCSharp SDK sample, but
When I remove bmdVideoInputEnableFormatDetection flag and apply bmdModeNTSC to IDeckLinkInput::EnableVideoInput(), It works so fine.

The another strange one is that BMDDisplayMode doesn't cause this problem in ohter C++ samples(CapturePreview, LoopThroughWithOpenGLCompositing...)
For more information, when I remove drawing code(like below), It works well(no choppy sound) on any BMDDisplayMode. It's same with the CapturePreviewCSharp sample.

Code: Select all


public void Draw()
{
    while (true)
    {
        var vPool = VideoFramePool.Pool.FirstOrDefault(x => x.Status == Enums.ProcessVideoFrameStatus.Converted);
        if (vPool == null)
        {
            Thread.Sleep(10);
            continue;
        }

        vPool.Status = Enums.ProcessVideoFrameStatus.Draw;
        var width = vPool.Frame.GetWidth();
        var height = vPool.Frame.GetHeight();
        vPool.Frame.GetBytes(out var bgrbytes);
        var rect = new Int32Rect(0, 0, width, height);

        Helpers.RunOnStaThread(() =>
        {
            Application.Current.Dispatcher.Invoke(() =>
            {
                if (bitmapSource == null)
                {
                    bitmapSource = new WriteableBitmap(width, height, DPI_X, DPI_Y, PIXEL_FORMAT, null);
                    var brush = new ImageBrush();
                    brush.ImageSource = bitmapSource;
                    this.PreviewImage.Background = brush;
                }

                if (bitmapSource.Width != width || bitmapSource.Height != height)
                {
                    bitmapSource = new WriteableBitmap(width, height, DPI_X, DPI_Y, PIXEL_FORMAT, null);
                    var brush = new ImageBrush();
                    brush.ImageSource = bitmapSource;
                    this.PreviewImage.Background = brush;
                }

                try
                {
                    bitmapSource.Lock();
                    var backBuffer = bitmapSource.BackBuffer;
                    var byteLen = width * height * 4;
                    unsafe
                    {
                        Buffer.MemoryCopy(bgrbytes.ToPointer(), backBuffer.ToPointer(), byteLen, byteLen);
                    }

                    bitmapSource.AddDirtyRect(rect);
                }
                catch (Exception ex)
                {
                }
                finally
                {
                    bitmapSource.Unlock();
                }
                Win32.uSleep(sleepUs);
            }
            );
        });
    }
}


Because of it, I suspected the performance issue at first, but it work well on bmdModeNTSC...

I just want to know why these are happened.
The DeckLink SDK Manual is so simple for it...
Our team have no experience for developing broadcast solutions actually.

[Cameron Nichols]

Hi Ray,

Could you explain what VideoFramePool.Pool is? Is this a queue of captured frames of IDeckLinkVideoInputFrame? Even having this code in your own thread may cause issue where input video frame allocation is exhausted and IDeckLinkInputCallback::VideoInputFrameArrived callback doesn't occur and frame and audio samples are dropped. You can test this theory by keeping log of IDeckLinkVideoInputFrame.GetStreamTime and IDeckLinkVideoInput.GetAvailableVideoFrameCount.

Looking at your code, there are a few processor intensive tasks, such as Buffer.MemoryCopy that will result in IDeckLinkVideoInputFrame objects being held longer. This is the likely reason why bmdModeNTSC capture is OK, but not other modes, as the pixel buffer size is smaller. Instead you should generate System.Drawing.Bitmap type without buffer copy, using constructor that accepts IntPtr pixel data. An example of this can be found in StillsCSharp sample.

That said, if your goal is to create your own screen preview, I suggest that you should instead be using IDeckLinkScreenPreviewCallback.DrawFrame callback instead of IDeckLinkInputCallback.VideoInputFrameArrived. The DrawFrame callback does not queue incoming frames, mitigating risk of the render task taking longer than a frame period. If you have not returned from the callback, subsequent frames are dropped. By implementing this, you then free your VideoInputFrameArrived callback to process audio samples, which you cannot drop.

Regards
Cameron