Video buffer in FileCapture sample grows uncontrollably

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

KerryL

  • Posts: 2
  • Joined: Fri Sep 03, 2021 3:33 pm
  • Real Name: Kerry Loux

Video buffer in FileCapture sample grows uncontrollably

PostTue Sep 07, 2021 2:53 pm

Hello all,

I'm using the DeckLink SDK 12.1 to interface to an UltraStudio 4K Mini. My goal is to save the incoming audio/video stream to file.

I started with the FileCapture sample application. I noticed that after beginning a recording, the PC's memory usage increases rapidly. I also noticed that if I stop the recording after 60 seconds, for example, the resulting video file only contains 30 seconds of data. The input stream is 2160p25.

After some investigation, I found that the SampleMemoryAllocator (the videoQueue member of the SinkWriter class) grows due to frames being queued faster than they are being written.

I used the Disk Speed Test application to check disk speeds, and it indicates that the read and write speeds should be sufficient for all tested video sizes/rates, including 4K 60 Hz.

I first made a modification to flush the queues at the end of the recording. I was surprised to see that as soon as recording stops, the rate at which the video samples are written to file slows to a crawl.

The modifications include:
1. Add the following public declarations to AudioSampleQueue.h:
Code: Select all
bool IsEmpty() const { return m_audioPacketQueue.empty(); }
   void Pop(IMFSample** sample);


2. Add the following public declarations to SampleMemoryAllocator.h:
Code: Select all
bool IsEmpty() const { return m_videoFrameQueue.empty(); }
   void Pop(IMFSample** sample);


3. Add the following to AudioSampleQueue.cpp:
Code: Select all
void AudioSampleQueue::Pop(IMFSample** sample)
{
   CComPtr<IMFSample>               audioSample;
   CComPtr<IDeckLinkAudioInputPacket>   audioPacket;
   CComPtr<IMFMediaBuffer>            sampleMediaBuffer;
   void*                        sampleBuffer;
   void*                        audioPacketBuffer;
   BMDTimeValue                  packetTime;
   uint32_t                     bufferLength;
   HRESULT                        hr = S_OK;

   if (m_audioPacketQueue.empty())
      return;

   {
      std::unique_lock<std::mutex> lock(m_audioPacketQueueMutex);
      audioPacket = m_audioPacketQueue.front();
      m_audioPacketQueue.pop();
   }

   if (!audioPacket)
      return;

   // Get pointer to audio buffer to copy to IMFMediaBuffer
   hr = audioPacket->GetBytes(&audioPacketBuffer);
   if (hr != S_OK)
      return;

   // Get time and duration of captured packet
   hr = audioPacket->GetPacketTime(&packetTime, kMFTimescale);
   if (hr != S_OK)
      return;

   bufferLength = audioPacket->GetSampleFrameCount() * (m_channelCount * (m_sampleBitDepth / 8));

   // Create a new memory buffer.
   hr = MFCreateMemoryBuffer(bufferLength, &sampleMediaBuffer);
   if (hr != S_OK)
      return;

   // Lock the buffer and copy the audio samples to the buffer.
   hr = sampleMediaBuffer->Lock((BYTE**)&sampleBuffer, NULL, NULL);
   if (hr != S_OK)
      return;

   memcpy(sampleBuffer, audioPacketBuffer, bufferLength);

   sampleMediaBuffer->Unlock();

   // Set the data length of the buffer.
   hr = sampleMediaBuffer->SetCurrentLength(bufferLength);
   if (hr != S_OK)
      return;

   // Create a media sample and add the buffer to the sample.
   hr = MFCreateSample(&audioSample);
   if (hr != S_OK)
      return;

   hr = audioSample->AddBuffer(sampleMediaBuffer);
   if (hr != S_OK)
      return;

   // Set the time stamp and the duration.
   hr = audioSample->SetSampleTime((LONGLONG)packetTime);
   if (hr != S_OK)
      return;

   hr = audioSample->SetSampleDuration((LONGLONG)audioPacket->GetSampleFrameCount() * kMFTimescale / bmdAudioSampleRate48kHz);

   *sample = audioSample.Detach();
}


4. Add the following to SampleMemoryAllocator.cpp:
Code: Select all
void SampleMemoryAllocator::Pop(IMFSample** sample)
{
   CComPtr<IMFSample>               videoSample;
   CComPtr<IDeckLinkVideoInputFrame>   videoFrame;
   void*                        videoBuffer;
   BMDTimeValue                  frameTime;
   BMDTimeValue                  frameDuration;
   HRESULT                        hr = S_OK;

   if (m_videoFrameQueue.empty())
      return;

   {
      std::unique_lock<std::mutex> lock(m_videoFrameQueueMutex);
      videoFrame = m_videoFrameQueue.front();
      m_videoFrameQueue.pop();
   }

   if (!videoFrame)
      return;

   // Get time and duration of captured frame
   hr = videoFrame->GetStreamTime(&frameTime, &frameDuration, kMFTimescale);
   if (hr != S_OK)
      return;

   // Get pointer to video buffer to match to IMFMediaBuffer
   hr = videoFrame->GetBytes(&videoBuffer);
   if (hr != S_OK)
      return;

   auto mediaBufferIter = m_allocatedBuffers.find(videoBuffer);
   if (mediaBufferIter == m_allocatedBuffers.end())
      // Did not find buffer
      return;

   // Found IMFMediaBuffer, unlock and construct IMFSample
   mediaBufferIter->second->Unlock();

   // Set the data length of the buffer.
   hr = mediaBufferIter->second->SetCurrentLength(videoFrame->GetRowBytes() * videoFrame->GetHeight());
   if (hr != S_OK)
      return;

   // Create a media sample and add the buffer to the sample.
   hr = MFCreateSample(&videoSample);
   if (hr != S_OK)
      return;

   hr = videoSample->AddBuffer(mediaBufferIter->second);
   if (hr != S_OK)
      return;

   // Set the time stamp and the duration.
   hr = videoSample->SetSampleTime((LONGLONG)frameTime);
   if (hr != S_OK)
      return;

   hr = videoSample->SetSampleDuration((LONGLONG)frameDuration);
   if (hr != S_OK)
      return;

   hr = videoSample->SetUINT32(MFSampleExtension_Interlaced, m_videoInterlaced);
   if (hr != S_OK)
      return;

   *sample = videoSample.Detach();
}


5. Modify SinkWriter::AudioSinkWriterThread and SinkWriter::VideoSinkWriterThread as follows:
Code: Select all
void SinkWriter::VideoSinkWriterThread(CComPtr<SampleMemoryAllocator> videoQueue)
{
   bool captureRunning = true;

   while (captureRunning)
   {
      bool               captureCancelled;
      CComPtr<IMFSample>      sample;

      if (!videoQueue->WaitForInputSample(&sample, captureCancelled) || captureCancelled)
      {
         captureRunning = false;
      }
      else
      {
         std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
         if (m_sinkWriter->WriteSample(m_videoStreamIndex, sample) != S_OK)
            captureRunning = false;
      }
   }

   while (!videoQueue->IsEmpty())
   {
      CComPtr<IMFSample> sample;
      videoQueue->Pop(&sample);
      std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
      if (m_sinkWriter->WriteSample(m_videoStreamIndex, sample) != S_OK)
         break;
   }
}

void SinkWriter::AudioSinkWriterThread(CComPtr<AudioSampleQueue> audioQueue)
{
   bool captureRunning = true;

   while (captureRunning)
   {
      bool               captureCancelled;
      CComPtr<IMFSample>      sample;
      
      if (!audioQueue->WaitForInputSample(&sample, captureCancelled) || captureCancelled)
      {
         captureRunning = false;
      }
      else
      {
         std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
         if (m_sinkWriter->WriteSample(m_audioStreamIndex, sample) != S_OK)
            captureRunning = false;
      }
   }

   while (!audioQueue->IsEmpty())
   {
      CComPtr<IMFSample> sample;
      audioQueue->Pop(&sample);
      std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
      if (m_sinkWriter->WriteSample(m_audioStreamIndex, sample) != S_OK)
         break;
   }
}


6. Modify CFileCaptureDlg::StopCapture() as follows:
Code: Select all
void CFileCaptureDlg::StopCapture()
{
   m_captureState = CaptureState::WaitingForValidInput;// Move this prior to the if() statement to prevent queuing new frames while flushing the queue

   if ((m_selectedDevice) && (m_selectedDevice->IsCapturing()))
      m_selectedDevice->StopCapture();

   UpdateInterface();
}


My modifications worked in terms of writing complete recordings and emptying the queues to reduce memory usage after the recording stopped. However, calls to m_sinkWriter->WriteSample() take a Really Long Time after recording has stopped. This also appears to be the bottleneck when writing while recording is active, but these calls are about 100x slower when recording has stopped (determined with a profiling tool).

So I have two questions:
1. Why is m_sinkWriter->WriteSample() so slow after recording has stopped? I expect it to take the same amount of time as when recording is continuing.
2. Any recommendations to be able to write samples fast enough to support real-time recording? I read on MSDN (not allowed to post the link?) that maybe the simplified Media Foundation encoding interface used in the example isn't appropriate for real-time recording. Why do other people use? Or are there settings I can try to improve performance?

Thanks,

Kerry
Offline

KerryL

  • Posts: 2
  • Joined: Fri Sep 03, 2021 3:33 pm
  • Real Name: Kerry Loux

Re: Video buffer in FileCapture sample grows uncontrollably

PostFri Sep 17, 2021 1:39 pm

Here's an update:

After making short recordings for two days, every one of which exhibited the issue described above, everything suddenly started working better. I'm not sure if I rebooted the PC or cycled power on the UltraStudio or both (or something else...) but suddenly I could record for several hours and the encoder keeps up (mostly).

But every once in a while, something would happen that would suddenly cause the queue to start growing just as I had seen originally. I continued to debug this, and found that it is being caused by an error in SampleMemoryAllocator::WaitForInputSample. It seems that occasionally, the videoBuffer pointer obtained in the videoFrame->GetBytes() call is not found within the m_allocatedBuffers map. In other words, we return false here:
Code: Select all
      auto mediaBufferIter = m_allocatedBuffers.find(videoBuffer);
      if (mediaBufferIter == m_allocatedBuffers.end())
         // Did not find buffer
         return false;


Because the buffers are added to the map with calls to SampleMemoryAllocator::AllocateBuffer from somewhere else inside the DeckLink libraries, I'm not sure what else I can do to fix this. Any ideas?

Thanks,

Kerry

Return to Software Developers

Who is online

Users browsing this forum: No registered users and 9 guests