- Posts: 2
- Joined: Fri Sep 03, 2021 3:33 pm
- Real Name: Kerry Loux
Hello all,
I'm using the DeckLink SDK 12.1 to interface to an UltraStudio 4K Mini. My goal is to save the incoming audio/video stream to file.
I started with the FileCapture sample application. I noticed that after beginning a recording, the PC's memory usage increases rapidly. I also noticed that if I stop the recording after 60 seconds, for example, the resulting video file only contains 30 seconds of data. The input stream is 2160p25.
After some investigation, I found that the SampleMemoryAllocator (the videoQueue member of the SinkWriter class) grows due to frames being queued faster than they are being written.
I used the Disk Speed Test application to check disk speeds, and it indicates that the read and write speeds should be sufficient for all tested video sizes/rates, including 4K 60 Hz.
I first made a modification to flush the queues at the end of the recording. I was surprised to see that as soon as recording stops, the rate at which the video samples are written to file slows to a crawl.
The modifications include:
1. Add the following public declarations to AudioSampleQueue.h:
2. Add the following public declarations to SampleMemoryAllocator.h:
3. Add the following to AudioSampleQueue.cpp:
4. Add the following to SampleMemoryAllocator.cpp:
5. Modify SinkWriter::AudioSinkWriterThread and SinkWriter::VideoSinkWriterThread as follows:
6. Modify CFileCaptureDlg::StopCapture() as follows:
My modifications worked in terms of writing complete recordings and emptying the queues to reduce memory usage after the recording stopped. However, calls to m_sinkWriter->WriteSample() take a Really Long Time after recording has stopped. This also appears to be the bottleneck when writing while recording is active, but these calls are about 100x slower when recording has stopped (determined with a profiling tool).
So I have two questions:
1. Why is m_sinkWriter->WriteSample() so slow after recording has stopped? I expect it to take the same amount of time as when recording is continuing.
2. Any recommendations to be able to write samples fast enough to support real-time recording? I read on MSDN (not allowed to post the link?) that maybe the simplified Media Foundation encoding interface used in the example isn't appropriate for real-time recording. Why do other people use? Or are there settings I can try to improve performance?
Thanks,
Kerry
I'm using the DeckLink SDK 12.1 to interface to an UltraStudio 4K Mini. My goal is to save the incoming audio/video stream to file.
I started with the FileCapture sample application. I noticed that after beginning a recording, the PC's memory usage increases rapidly. I also noticed that if I stop the recording after 60 seconds, for example, the resulting video file only contains 30 seconds of data. The input stream is 2160p25.
After some investigation, I found that the SampleMemoryAllocator (the videoQueue member of the SinkWriter class) grows due to frames being queued faster than they are being written.
I used the Disk Speed Test application to check disk speeds, and it indicates that the read and write speeds should be sufficient for all tested video sizes/rates, including 4K 60 Hz.
I first made a modification to flush the queues at the end of the recording. I was surprised to see that as soon as recording stops, the rate at which the video samples are written to file slows to a crawl.
The modifications include:
1. Add the following public declarations to AudioSampleQueue.h:
- Code: Select all
bool IsEmpty() const { return m_audioPacketQueue.empty(); }
void Pop(IMFSample** sample);
2. Add the following public declarations to SampleMemoryAllocator.h:
- Code: Select all
bool IsEmpty() const { return m_videoFrameQueue.empty(); }
void Pop(IMFSample** sample);
3. Add the following to AudioSampleQueue.cpp:
- Code: Select all
void AudioSampleQueue::Pop(IMFSample** sample)
{
CComPtr<IMFSample> audioSample;
CComPtr<IDeckLinkAudioInputPacket> audioPacket;
CComPtr<IMFMediaBuffer> sampleMediaBuffer;
void* sampleBuffer;
void* audioPacketBuffer;
BMDTimeValue packetTime;
uint32_t bufferLength;
HRESULT hr = S_OK;
if (m_audioPacketQueue.empty())
return;
{
std::unique_lock<std::mutex> lock(m_audioPacketQueueMutex);
audioPacket = m_audioPacketQueue.front();
m_audioPacketQueue.pop();
}
if (!audioPacket)
return;
// Get pointer to audio buffer to copy to IMFMediaBuffer
hr = audioPacket->GetBytes(&audioPacketBuffer);
if (hr != S_OK)
return;
// Get time and duration of captured packet
hr = audioPacket->GetPacketTime(&packetTime, kMFTimescale);
if (hr != S_OK)
return;
bufferLength = audioPacket->GetSampleFrameCount() * (m_channelCount * (m_sampleBitDepth / 8));
// Create a new memory buffer.
hr = MFCreateMemoryBuffer(bufferLength, &sampleMediaBuffer);
if (hr != S_OK)
return;
// Lock the buffer and copy the audio samples to the buffer.
hr = sampleMediaBuffer->Lock((BYTE**)&sampleBuffer, NULL, NULL);
if (hr != S_OK)
return;
memcpy(sampleBuffer, audioPacketBuffer, bufferLength);
sampleMediaBuffer->Unlock();
// Set the data length of the buffer.
hr = sampleMediaBuffer->SetCurrentLength(bufferLength);
if (hr != S_OK)
return;
// Create a media sample and add the buffer to the sample.
hr = MFCreateSample(&audioSample);
if (hr != S_OK)
return;
hr = audioSample->AddBuffer(sampleMediaBuffer);
if (hr != S_OK)
return;
// Set the time stamp and the duration.
hr = audioSample->SetSampleTime((LONGLONG)packetTime);
if (hr != S_OK)
return;
hr = audioSample->SetSampleDuration((LONGLONG)audioPacket->GetSampleFrameCount() * kMFTimescale / bmdAudioSampleRate48kHz);
*sample = audioSample.Detach();
}
4. Add the following to SampleMemoryAllocator.cpp:
- Code: Select all
void SampleMemoryAllocator::Pop(IMFSample** sample)
{
CComPtr<IMFSample> videoSample;
CComPtr<IDeckLinkVideoInputFrame> videoFrame;
void* videoBuffer;
BMDTimeValue frameTime;
BMDTimeValue frameDuration;
HRESULT hr = S_OK;
if (m_videoFrameQueue.empty())
return;
{
std::unique_lock<std::mutex> lock(m_videoFrameQueueMutex);
videoFrame = m_videoFrameQueue.front();
m_videoFrameQueue.pop();
}
if (!videoFrame)
return;
// Get time and duration of captured frame
hr = videoFrame->GetStreamTime(&frameTime, &frameDuration, kMFTimescale);
if (hr != S_OK)
return;
// Get pointer to video buffer to match to IMFMediaBuffer
hr = videoFrame->GetBytes(&videoBuffer);
if (hr != S_OK)
return;
auto mediaBufferIter = m_allocatedBuffers.find(videoBuffer);
if (mediaBufferIter == m_allocatedBuffers.end())
// Did not find buffer
return;
// Found IMFMediaBuffer, unlock and construct IMFSample
mediaBufferIter->second->Unlock();
// Set the data length of the buffer.
hr = mediaBufferIter->second->SetCurrentLength(videoFrame->GetRowBytes() * videoFrame->GetHeight());
if (hr != S_OK)
return;
// Create a media sample and add the buffer to the sample.
hr = MFCreateSample(&videoSample);
if (hr != S_OK)
return;
hr = videoSample->AddBuffer(mediaBufferIter->second);
if (hr != S_OK)
return;
// Set the time stamp and the duration.
hr = videoSample->SetSampleTime((LONGLONG)frameTime);
if (hr != S_OK)
return;
hr = videoSample->SetSampleDuration((LONGLONG)frameDuration);
if (hr != S_OK)
return;
hr = videoSample->SetUINT32(MFSampleExtension_Interlaced, m_videoInterlaced);
if (hr != S_OK)
return;
*sample = videoSample.Detach();
}
5. Modify SinkWriter::AudioSinkWriterThread and SinkWriter::VideoSinkWriterThread as follows:
- Code: Select all
void SinkWriter::VideoSinkWriterThread(CComPtr<SampleMemoryAllocator> videoQueue)
{
bool captureRunning = true;
while (captureRunning)
{
bool captureCancelled;
CComPtr<IMFSample> sample;
if (!videoQueue->WaitForInputSample(&sample, captureCancelled) || captureCancelled)
{
captureRunning = false;
}
else
{
std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
if (m_sinkWriter->WriteSample(m_videoStreamIndex, sample) != S_OK)
captureRunning = false;
}
}
while (!videoQueue->IsEmpty())
{
CComPtr<IMFSample> sample;
videoQueue->Pop(&sample);
std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
if (m_sinkWriter->WriteSample(m_videoStreamIndex, sample) != S_OK)
break;
}
}
void SinkWriter::AudioSinkWriterThread(CComPtr<AudioSampleQueue> audioQueue)
{
bool captureRunning = true;
while (captureRunning)
{
bool captureCancelled;
CComPtr<IMFSample> sample;
if (!audioQueue->WaitForInputSample(&sample, captureCancelled) || captureCancelled)
{
captureRunning = false;
}
else
{
std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
if (m_sinkWriter->WriteSample(m_audioStreamIndex, sample) != S_OK)
captureRunning = false;
}
}
while (!audioQueue->IsEmpty())
{
CComPtr<IMFSample> sample;
audioQueue->Pop(&sample);
std::lock_guard<std::mutex> lock(m_sinkWriterMutex);
if (m_sinkWriter->WriteSample(m_audioStreamIndex, sample) != S_OK)
break;
}
}
6. Modify CFileCaptureDlg::StopCapture() as follows:
- Code: Select all
void CFileCaptureDlg::StopCapture()
{
m_captureState = CaptureState::WaitingForValidInput;// Move this prior to the if() statement to prevent queuing new frames while flushing the queue
if ((m_selectedDevice) && (m_selectedDevice->IsCapturing()))
m_selectedDevice->StopCapture();
UpdateInterface();
}
My modifications worked in terms of writing complete recordings and emptying the queues to reduce memory usage after the recording stopped. However, calls to m_sinkWriter->WriteSample() take a Really Long Time after recording has stopped. This also appears to be the bottleneck when writing while recording is active, but these calls are about 100x slower when recording has stopped (determined with a profiling tool).
So I have two questions:
1. Why is m_sinkWriter->WriteSample() so slow after recording has stopped? I expect it to take the same amount of time as when recording is continuing.
2. Any recommendations to be able to write samples fast enough to support real-time recording? I read on MSDN (not allowed to post the link?) that maybe the simplified Media Foundation encoding interface used in the example isn't appropriate for real-time recording. Why do other people use? Or are there settings I can try to improve performance?
Thanks,
Kerry