DX9 Frame + VANC on SDI Output

Hi,

We are working on an application that takes a DX9 frame, add Ancillary data to it and outputs it through the SDI on a DeckLink Studio 4K.

We are using the Blackmagic DeckLink SDK 10.8.3

We used the VancOuput sample project to do some initial tests and started modifying it to reach our goals.

Basically we add the IDeckLinkDX9ScreenPreviewHelper Interface steps described into SDK Documentation. But we don't find the connection between both... How to take a DX9 frame to a VideoFrame, to be able to add the ancillary data and to play it back.

We are looking for some sample code on how to output a DX9 frame on the SDI.

Attached is the code we have for now.

To make it shorter, here is basically the main method overview, we are able to create all object/interfaces, no errors... the problem here, i would say, is plumbing =D

Code: Select all

   
   //Initialize COM object;

   //Create an IDeckLinkDX9ScreenPreviewHelper object interface using CoCreateInstance.
   
   //Setup Direct 3D parameters

   //Create Direct3D9 Device

   //Initialize Direct3D9 Device trhough DeckLinkScreenPreviewHelper
   
   // Create an IDeckLinkIterator object to enumerate all DeckLink cards in the system
      
   // Obtain the first DeckLink device
      
   // Obtain the output interface for the DeckLink device
   
   // Create an instance of ScreenPreview callback
   
   // Create an instance of output callback
   
   // Set the DeckLinkScreenPreviewCallbackobject to the DeckLink device's output interface
   
   // Set the OutputCallback object to the DeckLink device's output interface
   
   // Enable video output
   
   // Create a frame with defined format (This is from the original code, theoretically it should be removed)
   
   // Schedule a blue frame 3 times (This is from the original code, theoretically it should be removed)
   
   //Tries to print something into the Direct3D9 Device

   // Start the playback
   
   // Wait until user presses Enter to exit the application.


Complete Code:

Code: Select all

#include "platform.h"
#include <windowsx.h>
#include <Windows.h>
#include <stdio.h>
#include <iostream>
#include <d3d9.h>
#include "DXSDK-June2010/include/d3dx9.h"
#include "DXSDK-June2010/include/DxErr.h"
#pragma comment(lib, "../DXSDK-June2010/lib/dxerr.lib")

int (WINAPIV * __vsnprintf)(char *, size_t, const char*, va_list) = _vsnprintf;

// Video mode parameters
const BMDDisplayMode      kDisplayMode = bmdModeHD1080i50;
const BMDVideoOutputFlags kOutputFlag  = bmdVideoOutputVANC;
const BMDPixelFormat      kPixelFormat = bmdFormat10BitYUV;

// Frame parameters
const INT32_UNSIGNED kFrameDuration = 1000;
const INT32_UNSIGNED kTimeScale = 25000;
const INT32_UNSIGNED kFrameWidth = 1920;
const INT32_UNSIGNED kFrameHeight = 1080;
const INT32_UNSIGNED kRowBytes = 5120;

// 10-bit YUV blue pixels
const INT32_UNSIGNED kBlueData[4] = { 0x40aa298, 0x2a8a62a8, 0x298aa040, 0x2a8102a8 };

// Studio Camera control packet:
// Set dynamic range to film.
// See Studio Camera manual for more information on protocol.
const INT8_UNSIGNED kSDIRemoteControlData[9] = { 0x00, 0x07, 0x00, 0x00, 0x01, 0x07, 0x01, 0x00, 0x00 };

// Data Identifier
const INT8_UNSIGNED kSDIRemoteControlDID = 0x51;

// Secondary Data Identifier
const INT8_UNSIGNED kSDIRemoteControlSDID = 0x53;

// Define VANC line for camera control
const INT32_UNSIGNED kSDIRemoteControlLine = 16;

// Keep track of the number of scheduled frames
INT32_UNSIGNED gTotalFramesScheduled = 0;

class OutputCallback: public IDeckLinkVideoOutputCallback
{
public:
   OutputCallback(IDeckLinkOutput* deckLinkOutput)
   {
      m_deckLinkOutput = deckLinkOutput;
      m_deckLinkOutput->AddRef();
   }
   virtual ~OutputCallback(void)
   {
      m_deckLinkOutput->Release();
   }
   HRESULT   STDMETHODCALLTYPE ScheduledFrameCompleted(IDeckLinkVideoFrame* completedFrame, BMDOutputFrameCompletionResult result)
   {
      // When a video frame completes,reschedule another frame
      m_deckLinkOutput->ScheduleVideoFrame(completedFrame, gTotalFramesScheduled*kFrameDuration, kFrameDuration, kTimeScale);
      gTotalFramesScheduled++;
      return S_OK;
   }
   
   HRESULT   STDMETHODCALLTYPE ScheduledPlaybackHasStopped(void)
   {
      return S_OK;
   }
   // IUnknown needs only a dummy implementation
   HRESULT   STDMETHODCALLTYPE QueryInterface (REFIID iid, LPVOID *ppv)
   {
      return E_NOINTERFACE;
   }
   
   ULONG STDMETHODCALLTYPE AddRef()
   {
      return 1;
   }
   
   ULONG STDMETHODCALLTYPE Release()
   {
      return 1;
   }
private:
   IDeckLinkOutput*  m_deckLinkOutput;
};

class DeckLinkScreenPreviewCallback : public IDeckLinkScreenPreviewCallback
{
public:
   DeckLinkScreenPreviewCallback(IDeckLinkDX9ScreenPreviewHelper* screenPreviewHelper)
   {
      m_screenPreviewHelper = screenPreviewHelper;
      m_screenPreviewHelper->AddRef();
   }
   virtual ~DeckLinkScreenPreviewCallback(void)
   {
      m_screenPreviewHelper->Release();
   }
   HRESULT   STDMETHODCALLTYPE DrawFrame(IDeckLinkVideoFrame* theFrame)
   {
      m_screenPreviewHelper->SetFrame(theFrame);
      
      return S_OK;
   }

   // IUnknown needs only a dummy implementation
   HRESULT   STDMETHODCALLTYPE QueryInterface(REFIID iid, LPVOID *ppv)
   {
      return E_NOINTERFACE;
   }

   ULONG STDMETHODCALLTYPE AddRef()
   {
      return 1;
   }

   ULONG STDMETHODCALLTYPE Release()
   {
      return 1;
   }
private:
   IDeckLinkDX9ScreenPreviewHelper*  m_screenPreviewHelper;
};

//  This function translates a byte into a 10-bit sample
//   x x x x x x x x x x x x
//       -------------------
//   | | |  0-7 raw data   |
//   | |
//   | even parity bit
//   inverse of bit 8
static inline INT32_UNSIGNED EncodeByte(INT32_UNSIGNED byte)
{
   INT32_UNSIGNED temp = byte;
   // Calculate the even parity bit of bits 0-7 by XOR every individual bits
   temp ^= temp >> 4;
   temp ^= temp >> 2;
   temp ^= temp >> 1;
   // Use lsb as parity bit
   temp &= 1;
   // Put even parity bit on bit 8
   byte |= temp << 8;
   // Bit 9 is inverse of bit 8
   byte |= ((~temp) & 1) << 9;
   return byte;
}
// This function writes 10bit ancillary data to 10bit luma value in YUV 10bit structure
static void WriteAncDataToLuma(INT32_UNSIGNED*& sdiStreamPosition, INT32_UNSIGNED value, INT32_UNSIGNED dataPosition)
{
   switch(dataPosition % 3)
   {
      case 0:
         *sdiStreamPosition++  = (value) << 10;
         break;
      case 1:
         *sdiStreamPosition = (value);
         break;
      case 2:
         *sdiStreamPosition++ |= (value) << 20;
         break;
      default:
         break;
   }
}

static void WriteAncillaryDataPacket(INT32_UNSIGNED* line, const INT8_UNSIGNED did, const INT8_UNSIGNED sdid, const INT8_UNSIGNED* data, INT32_UNSIGNED length)
{
   // Sanity check
   if (length == 0 || length > 255)
      return;
   
   const INT32_UNSIGNED encodedDID  = EncodeByte(did);
   const INT32_UNSIGNED encodedSDID = EncodeByte(sdid);
   const INT32_UNSIGNED encodedDC   = EncodeByte(length);
   
   // Start sequence
   *line++ = 0;
   *line++ = 0x3ff003ff;
   
   // DID
   *line++ = encodedDID << 10;
   
   // SDID and DC
   *line++ = encodedSDID | (encodedDC << 20);
   
   // Checksum does not include the start sequence
   INT32_UNSIGNED sum = encodedDID + encodedSDID + encodedDC;
   // Write the payload
   for(INT32_UNSIGNED i = 0; i < length; ++i)
   {
      const INT32_UNSIGNED encoded = EncodeByte(data[i]);
      WriteAncDataToLuma(line, encoded, i);
      sum += encoded & 0x1ff;
   }
   
   // Checksum % 512 then copy inverse of bit 8 to bit 9
   sum &= 0x1ff;
   sum |= ((~(sum << 1)) & 0x200);
   WriteAncDataToLuma(line, sum, length);
}

static void SetVancData(IDeckLinkVideoFrameAncillary* ancillary)
{
   HRESULT   result;
   INT32_UNSIGNED* buffer;
   
   result = ancillary->GetBufferForVerticalBlankingLine(kSDIRemoteControlLine, (void **)&buffer);
   if (result != S_OK)
   {
      fprintf(stderr, "Could not get buffer for Vertical blanking line - result = %08x\n", result);
      return;
   }
   // Write camera control data to buffer
   WriteAncillaryDataPacket(buffer, kSDIRemoteControlDID, kSDIRemoteControlSDID, kSDIRemoteControlData, sizeof(kSDIRemoteControlData)/sizeof(kSDIRemoteControlData[0]));
}

static void FillBlue(IDeckLinkMutableVideoFrame* theFrame)
{
   INT32_UNSIGNED* nextWord;
   INT32_UNSIGNED  wordsRemaining;
   
   theFrame->GetBytes((void**)&nextWord);
   wordsRemaining = (kRowBytes * kFrameHeight) / 4;
   
   while (wordsRemaining > 0)
   {
      *(nextWord++) = kBlueData[0];
      *(nextWord++) = kBlueData[1];
      *(nextWord++) = kBlueData[2];
      *(nextWord++) = kBlueData[3];
      wordsRemaining = wordsRemaining - 4;
   }
}

static IDeckLinkMutableVideoFrame* CreateFrame(IDeckLinkOutput* deckLinkOutput)
{
   HRESULT                         result;
   IDeckLinkMutableVideoFrame*     frame = NULL;
   IDeckLinkVideoFrameAncillary*   ancillaryData = NULL;
   
   result = deckLinkOutput->CreateVideoFrame(kFrameWidth, kFrameHeight, kRowBytes, kPixelFormat, bmdFrameFlagDefault, &frame);
   if (result != S_OK)
   {
      fprintf(stderr, "Could not create a video frame - result = %08x\n", result);
      goto bail;
   }
   FillBlue(frame);
   result = deckLinkOutput->CreateAncillaryData(kPixelFormat, &ancillaryData);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not create Ancillary data - result = %08x\n", result);
      goto bail;
   }
   SetVancData(ancillaryData);
   result = frame->SetAncillaryData(ancillaryData);
   if (result != S_OK)
   {
      fprintf(stderr, "Fail to set ancillary data to the frame - result = %08x\n", result);
      goto bail;
   }
bail:
   // Release the Ancillary object
   if(ancillaryData != NULL)
      ancillaryData->Release();
   return frame;
}

int main(int argc, const char * argv[])
{
   
   IDeckLinkIterator*      deckLinkIterator = NULL;
   IDeckLink*              deckLink         = NULL;
   IDeckLinkOutput*        deckLinkOutput   = NULL;
   OutputCallback*         outputCallback   = NULL;
   DeckLinkScreenPreviewCallback* deckLinkScreenPreviewCallback = NULL;
   IDeckLinkVideoFrame*    videoFrameBlue   = NULL;
   HRESULT                 result;
   
   IDeckLinkDX9ScreenPreviewHelper* deckLinkScreenPreviewHelper = NULL;
   IDirect3D9Ex* d3dEx = NULL;
   IDirect3DDevice9* dxDevice = NULL;

   HWND hwnd = NULL;
   //HWND hwnd = CreateWindowA("STATIC", "dummy", NULL, 0, 0, 100, 100, NULL, NULL, NULL, NULL);

   Initialize();

   //Create an IDeckLinkDX9ScreenPreviewHelper object interface using CoCreateInstance.
   result = CoCreateInstance(CLSID_CDeckLinkDX9ScreenPreviewHelper, NULL, CLSCTX_ALL, IID_IDeckLinkDX9ScreenPreviewHelper, (void**)&deckLinkScreenPreviewHelper);
   if (result != S_OK)
   {
      fprintf(stderr, "Could not create an instance of IDeckLinkDX9ScreenPreviewHelper. The DeckLink drivers may not be installed\n");
      goto bail;
   }
   else
   {
      fprintf(stderr, "Instance of IDeckLinkDX9ScreenPreviewHelper successfully created\n");
   }

   //Setup Direct 3D parameters :
   D3DPRESENT_PARAMETERS d3dpp;
   ZeroMemory(&d3dpp, sizeof(d3dpp));
   d3dpp.Windowed = TRUE;
   d3dpp.BackBufferCount = 1;
   d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
   d3dpp.BackBufferWidth = 200;
   d3dpp.BackBufferHeight = 200;
   d3dpp.BackBufferFormat = D3DFMT_A8R8G8B8;
   d3dpp.hDeviceWindow = NULL;
   d3dpp.PresentationInterval = D3DPRESENT_INTERVAL_IMMEDIATE;

   result = Direct3DCreate9Ex(D3D_SDK_VERSION, (IDirect3D9Ex **)&d3dEx);
   if (result != S_OK)
   {
      fprintf(stderr, "Could not create an instance of IDirect3D9Ex - result = %08x\n", result);
      goto bail;
   }
   else
   {
      fprintf(stderr, "Instance of IDirect3D9Ex succesfully created\n");
   }

   result = d3dEx->CreateDevice(D3DADAPTER_DEFAULT, D3DDEVTYPE_HAL, NULL, D3DCREATE_HARDWARE_VERTEXPROCESSING | D3DCREATE_MULTITHREADED | D3DCREATE_PUREDEVICE, &d3dpp, &dxDevice);
   if (FAILED(result)) {
      fprintf(stderr, "Unable to create the Direct3D device. result = %08x\n", result);
      std::cerr << "Error: " << DXGetErrorString(result) << ", description: " << DXGetErrorDescription(result) << std::endl;
   }
   else
   {
      fprintf(stderr, "Direct3D device created.\n");
   }

   result = deckLinkScreenPreviewHelper->Initialize(dxDevice);
   if (FAILED(result)) {
      fprintf(stderr, "Unable to initialize the Direct3D device. result = %08x\n", result);
      //fprintf(stderr, "Error: %s error description: %s\n", DXGetErrorString(result), DXGetErrorDescription(result));
      std::cerr << "Error: " << DXGetErrorString(result) << ", description: " << DXGetErrorDescription(result) << std::endl;
   }
   else
   {
      fprintf(stderr, "Direct3D device initialized.\n");
   }

   
   // Create an IDeckLinkIterator object to enumerate all DeckLink cards in the system
   result = GetDeckLinkIterator(&deckLinkIterator);
   if(result != S_OK)
   {
      fprintf(stderr, "A DeckLink iterator could not be created.  The DeckLink drivers may not be installed.\n");
      goto bail;
   }
   
   // Obtain the first DeckLink device
   result = deckLinkIterator->Next(&deckLink);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not find DeckLink device - result = %08x\n", result);
      goto bail;
   }
   
   // Obtain the output interface for the DeckLink device
   result = deckLink->QueryInterface(IID_IDeckLinkOutput, (void**)&deckLinkOutput);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not obtain the IDeckLinkInput interface - result = %08x\n", result);
      goto bail;
   }
   
   // Create an instance of ScreenPreview callback
   deckLinkScreenPreviewCallback = new DeckLinkScreenPreviewCallback(deckLinkScreenPreviewHelper);
   if(deckLinkScreenPreviewCallback == NULL)
   {
      fprintf(stderr, "Could not create deckLinkScreenPreview callback object\n");
      goto bail;   
   }

   // Create an instance of output callback
   outputCallback = new OutputCallback(deckLinkOutput);
   if (outputCallback == NULL)
   {
      fprintf(stderr, "Could not create output callback object\n");
      goto bail;
   }

   // Set the callback object to the DeckLink device's output interface
   result = deckLinkOutput->SetScreenPreviewCallback(deckLinkScreenPreviewCallback);
   if (result != S_OK)
   {
      fprintf(stderr, "Could not set callback - result = %08x\n", result);
      goto bail;
   }
   
   // Set the callback object to the DeckLink device's output interface
   result = deckLinkOutput->SetScheduledFrameCompletionCallback(outputCallback);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not set callback - result = %08x\n", result);
      goto bail;
   }
   
   // Enable video output
   result = deckLinkOutput->EnableVideoOutput(kDisplayMode, kOutputFlag);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not enable video output - result = %08x\n", result);
      goto bail;
   }
   
   // Create a frame with defined format
   videoFrameBlue = CreateFrame(deckLinkOutput);
   
   // Schedule a blue frame 3 times
   for(int i = 0; i < 3; i++)
   {
      result = deckLinkOutput->ScheduleVideoFrame(videoFrameBlue, gTotalFramesScheduled*kFrameDuration, kFrameDuration, kTimeScale);
      if(result != S_OK)
      {
         fprintf(stderr, "Could not schedule video frame - result = %08x\n", result);
         goto bail;
      }
      gTotalFramesScheduled ++;
   }
   
   dxDevice->DrawPrimitive(D3DPT_TRIANGLELIST, 0, 1);
   dxDevice->BeginScene();
   dxDevice->Clear(0, NULL, D3DCLEAR_TARGET, D3DCOLOR_XRGB(0, 255, 0), 1.0f, 0);
   deckLinkScreenPreviewHelper->Render(NULL);
   dxDevice->EndScene();
   dxDevice->Present(NULL, NULL, NULL, NULL);

   // Start
   result = deckLinkOutput->StartScheduledPlayback(0, kTimeScale, 1.0);
   if(result != S_OK)
   {
      fprintf(stderr, "Could not start - result = %08x\n", result);
      goto bail;
   }
   
   // Wait until user presses Enter
   printf("Monitoring... Press <RETURN> to exit\n");
   
   getchar();
   
   printf("Exiting.\n");
   
   // Stop capture
   result = deckLinkOutput->StopScheduledPlayback(0, NULL, 0);
   
   // Disable the video input interface
   result = deckLinkOutput->DisableVideoOutput();
   
   // Release resources
bail:
   
   // Wait until user presses Enter
   printf("Bail: Error handling... Press <RETURN> to exit\n");

   getchar();

   printf("Exiting.\n");

   // Release the video input interface
   if(deckLinkOutput != NULL)
      deckLinkOutput->Release();
   
   // Release the Decklink object
   if(deckLink != NULL)
      deckLink->Release();
   
   // Release the DeckLink iterator
   if(deckLinkIterator != NULL)
      deckLinkIterator->Release();
   
   // Release the videoframe object
   if(videoFrameBlue != NULL)
      videoFrameBlue->Release();
   
   // Release the outputCallback callback object
   if(outputCallback)
      delete outputCallback;
   
   return(result == S_OK) ? 0 : 1;
}


Hi, i want to do the same Blue frame in 720p or in SD, how to do the caculation of the word ?

I mean this word : const uint32_t kBlueData[4] = { 0x40aa298, 0x2a8a62a8, 0x298aa040, 0x2a8102a8 };

I need to calculate it for SD or 720p

Thanks

[Cameron Nichols]

Hi Cladio,

I would try using the IDeckLinkInputCallback::VideoInputFrameArrived callback, rather than IDeckLinkScreenPreviewCallback to receive your frame. Although both can provide input frames, the VideoInputFrameArrived is guaranteed to provide every frame, whereas the screen preview helper will only provide as many frames as your DX9 render task can keep up with.

Reviewing the existing codebase, I cannot see any path from DeckLinkScreenPreviewCallback::DrawFrame to your output path. It is possible here to take the incoming IDeckLinkVideoFrame, attach your required Ancillary data and reschedule the same frame with IDeckLinkOutput::ScheduleVideoFrame. But I would do this into VideoInputFrameArrived callback as discussed above.

Also, I see your DX9 rendering (the bit enclosed by BeginScene and EndScene/Present), appears to be done in main(), please review the CapturePreviewCSharp SDK sample to see the correct usage of these calls.

Finally, if you are able to migrate your application to Desktop Video SDK 10.10 or later, then you can take advantage of the newer VANC interfaces, which simplifies embedding ST 291 Type 2 ancillary packets in output frames.

Regards
Cameron