Decklink | modify inputstream frame-by-frame

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

grekat

  • Posts: 2
  • Joined: Mon Apr 05, 2021 10:16 am
  • Real Name: Katharina Greiner

Decklink | modify inputstream frame-by-frame

PostMon Apr 05, 2021 10:41 am

Hello!

I´m new here and using the BMD Decklink SDK.

For a project we would like to modify the incoming inputstream of a Decklink 8K Pro by using a mathematical calculation or a LUT and send the modified stream to the output.
Our plan is to access the current frame frame-by-frame, recalculate the frame and send it to the output of the Decklink. We used for our attempts the examples "VancCapture", "SychronisedCapture" and "FileCapture", but we can´t finde a solution to solve our this.

If somebody knews a solution for this, we would be very thankful. Thanks!
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: Decklink | modify inputstream frame-by-frame

PostFri Apr 09, 2021 4:16 am

Hi Katharina,

Please have a look at the Desktop Video SDK sample InputLoopThrough (Desktop Video SDK 11.5 and later), which demonstrates rescheduling of input frames for output. It also performs a number of other tasks such as measuring latency, but you can ignore these functions.

You can modify the method processVideo in InputLoopThrough.cpp to add your custom frame processing. The input parameter videoFrame provides you the captured frame. You can access the underlying frame data by calling IDeckLinkVideoFrame::GetBytes() with videoFrame->getVideoFramePtr(). Once you have finished modifying the data, you can queue the frame for output by calling DeckLinkOutputDevice::scheduleVideoFrame, as shown in sample.

The processVideo call is performed with a dispatch queue, allowing for concurrent processing of frames on multiple threads. This is important for when your processing is longer than 1 frame period, ensuring that your scheduled frames can keep with the playback rate. You can adjust the number of processing threads by modifying constant kVideoDispatcherThreadCount.

Regards
Cameron
Offline

grekat

  • Posts: 2
  • Joined: Mon Apr 05, 2021 10:16 am
  • Real Name: Katharina Greiner

Re: Decklink | modify inputstream frame-by-frame

PostFri Apr 30, 2021 10:16 am

Thank you, Cameron for your help.

Following your explanations on how to change the InputLoopThrough-example we managed to get our pipeline to work like this:

10Bit YUV SDI input ----> [custom transformations] -----> 10Bit YUV SDI output

Our input format via SDI is 10Bit YUV as well as our output format to the monitor connected via SDI. We are now able to do our own color operations.
However the processing that we need to implement needs to be done in RGB space. Right now we are doing a manual pixel-by-pixel conversion 10BitYUV->10BitRGB and back with a for-loop. This is currently working but not very performant.

Would the DeckLink 8K Pro be able to directly provide a hardware-converted signal in 10Bit RGB, independent from the actual SDI-Input format?
Would it be possible to change the InputLoopThrough example, so the `std::shared_ptr<LoopThroughVideoFrame>& videoFrame` parameter of the `processVideo` function would already be in 10Bit RGB regardless of the original SDI-input Pixel format?

Desired Pipeline:

10Bit YUV SDI input --> hardware converted to 10Bit RGB --> [custom transformations] --> hardware converted to 10Bit YUV --> 10Bit YUV SDI output


We've also found the API function `IDeckLinkVideoConversion::ConvertFrame` in the SDK manual at 2.4.14. However we didn't yet manage to get a frame converted. It also didn't become quite clear to us if this SDK function would be hardware accellerated by the DeckLink 8K Pro.
So we first tried to create a new 10Bit RGB VideoFrame inside the loopthrough example filled with black color to see if the DeckLink 8K Pro would directly accept scheduling a RGB frame. But we didn't get the frame scheduled because of our lack of knowledge about com_ptr().

Which would be the best way to have a performant RGB pipeline regardless of the actual SDI input and output?

Thanks a lot for your help!
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: Decklink | modify inputstream frame-by-frame

PostFri May 07, 2021 1:18 am

Hi Katharina,

To add conversion to r210 in InputLoopThrough sample, please make the following change to processVideo method:
Code: Select all
void processVideo(std::shared_ptr<LoopThroughVideoFrame>& videoFrame, com_ptr<DeckLinkOutputDevice>& deckLinkOutput)
{
    // ...
    if (!deckLinkOutput->isPlaybackActive())
        return;

    // Convert frame to r210 if required
    if (videoFrame->getVideoFramePtr()->GetPixelFormat() != bmdFormat10BitRGB)
    {
        // Convert frame to r210 format
        com_ptr<IDeckLinkMutableVideoFrame> convertedVideoFrame;
        com_ptr<IDeckLinkVideoConversion> frameConverter;
      
        frameConverter = CreateVideoConversionInstance();
        if (!frameConverter)
            return;
      
        if (deckLinkOutput->getDeckLinkOutput()->CreateVideoFrame(
            (int32_t)videoFrame->getVideoFramePtr()->GetWidth(),
            (int32_t)videoFrame->getVideoFramePtr()->GetHeight(),
            (int32_t)((videoFrame->getVideoFramePtr()->GetWidth() + 63) / 64) * 256, // See 3.4 Pixel Formats
            bmdFormat10BitRGB,
            videoFrame->getVideoFramePtr()->GetFlags(),
            convertedVideoFrame.releaseAndGetAddressOf()) != S_OK)
        {
            return;
        }
      
        if (frameConverter->ConvertFrame(videoFrame->getVideoFramePtr(), convertedVideoFrame.get()) != S_OK)
            return;
      
        videoFrame->setVideoFrame(com_ptr<IDeckLinkVideoFrame>(IID_IDeckLinkVideoFrame, convertedVideoFrame));
    }
   
    // Simulate doing something by using a busy wait loop
    // ...

    // At end of function, remember to queue your output frame
    deckLinkOutput->scheduleVideoFrame(std::move(videoFrame));
}

Regards
Cameron

Return to Software Developers

Who is online

Users browsing this forum: No registered users and 17 guests