ScheduleAudioSamples

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

ScheduleAudioSamples

PostTue Jun 19, 2018 8:18 pm

I'm not sure how to use ScheduleAudioSamples to playback sound synched to 720p5994 video.

If I put ScheduleAudioSamples inside RenderAudioSamples, does it get called every time a new video frame is loaded?

Should my audio buffer contain one video frame's worth of samples (i.e. 800)?
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostTue Jun 19, 2018 10:41 pm

Hi Jonas,

Yes, when calling ScheduleAudioSamples you should schedule an equivalent of 1 video frame worth of samples.

So for 720p5994,
48KHz / (60000 / 1001)
= 800.8

Because it is not an integral number, you need to call ScheduleAudioSamples with buffers of 801, 801, 801, 801, 800 samples to match video frame rate.

Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostWed Jun 20, 2018 2:18 pm

Hi Cameron,

Thank you for clarifying. I can't quite figure out how the streamTime and timeScale arguments work.
Code: Select all
output->ScheduleAudioSamples(audioBuffer, samplesPerFrame, streamTime, timeScale, NULL);

Before everytime this function gets called, should I do the following:
- set audioBuffer to point to the next 800 or 801 samples
- set samplesPerFrame to 800 or 801
- increment streamTime by the amount of timeScale (???)
- set timeScale to 60000ms/samplesPerFrame (???)
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostWed Jun 20, 2018 9:09 pm

For a test, I filled audioBuffer with 800 samples. I want to repeatedly schedule the same samples to play with each frame of video. However, I cannot figure out what to use for the streamTime and timeScale parameters that will cause the sound play smoothly with out breaks. Instead I always get a pulsating sound.
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostThu Jun 21, 2018 5:19 am

Hi Jonas,

The timeScale should be 48000, and streamTime should be an incremental time (0, 801, 1602, ...). But if you set flag bmdAudioOutputStreamContinuous in IDeckLinkOutput::EnableAudioOutput, then you can set both timeScale and streamTime to 0 and it will be handled as a continuous buffer.

BTW - Are you monitoring the size of your Audio buffer? The IDeckLinkAudioOutputCallback::RenderAudioSamples callback occurs at 50Hz, when inside the callback, call IDeckLinkOutput::GetBufferedAudioSampleFrameCount to get the current buffer size.

Because you have a frame rate of 59.94fps, then you will need to regularly schedule 2 frames worth of audio samples inside the callback, otherwise you won't retain your preroll size and exhaust the scheduled audio samples.

Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostThu Jun 21, 2018 2:27 pm

Thanks again Cameron.

Unfortunately, I'm still very confused, and there doesn't seem to be a complete example that comes with the SDK.

Currently, I don't call GetBufferedAudioSampleFrameCount. I saw this in the docs but it's not clear to me what I would with the value returned. Also, I find the 50Hz rate confusing. If I'm scheduling enough audio samples for 1 video frame, I'm not sure how to code for the 50 Hz.

Does using GetBufferedAudioSampleFrameCount help schedule the 2 frames of audio samples?

I think I have all the bits and pieces to make this work. It's just not clear from the docs how to put them all together.

Jonas
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostThu Jun 21, 2018 3:25 pm

Another question...If I use bmdAudioOutputStreamContinuous is it still possible to sync the audio with the video frames?

I did a small test using bmdAudioOutputStreamContinuous with the following code for scheduling. This sets sampleSize to 801, 801, 801, 801, 800, etc.
Code: Select all
int sampleSize = audioSamplesPerFrame + (totalAudioSamplesScheduled % 5 == 4 ? 0 : 1);
HRESULT hresult = output->ScheduleAudioSamples(audioBuffer, sampleSize, 0, 0, NULL);
if (hresult == S_OK)
   totalAudioSamplesScheduled++;

There is an improvement in the audio output. I now hear an almost continuous signal. There is still a small gap of no sound that repeats. I wonder if this is because I am not taking into account the 50 Hz callback rate. Is this what I need to use GetBufferedAudioSampleFrameCount for?
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostThu Jun 21, 2018 3:33 pm

This didn't seem to help, but am I on the right track?

Code: Select all
unsigned int bufferedSampleCount;
output->GetBufferedAudioSampleFrameCount(&bufferedSampleCount);

int sampleSize = audioSamplesPerFrame + (totalAudioSamplesScheduled % 5 == 4 ? 0 : 1);
HRESULT hresult = output->ScheduleAudioSamples(audioBuffer, sampleSize - bufferedSampleCount, 0, 0, NULL);

if (hresult == S_OK)
   totalAudioSamplesScheduled++;

Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostFri Jun 22, 2018 5:15 am

Hi Jonas,

Just checking - you have prerolled your audio buffer before starting playback? The audio preroll size should match the number of video frames prerolled.

The code you provided looks fine to me. You still need to consider where you need to call ScheduleAudioSamples with 2 video frames worth of audio samples once in every approx 5 IDeckLinkAudioOutputCallback:: RenderAudioSamples callbacks to compensate for 59.94fps vs 50Hz.

By monitoring the bufferedSampleCount value, you should be able to maintain the audio buffer size to the preroll size. Likely that the blanks in audio are that your scheduled audio buffer is being exhausted.

Finally, if your audio/video samples are being generated separately, you may find it easier just to buffer 48kHz/50Hz = fixed 960 samples each callback. You still need to match preroll video and audio however, otherwise your audio/video will be out of sync.

Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostFri Jun 22, 2018 9:29 pm

Is there a code example of this? I have so many questions. For instance, how do I sync the video and the audio preroll? Can I schedule several frames of preroll audio at one time?
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostMon Jun 25, 2018 8:13 am

Hi Jonas,

Please have a look at the SignalGenerator (macOS, Windows, Linux) or the TestPattern (Linux) SDK samples. Both samples preroll 3 frames of video and also audio.

Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostMon Jun 25, 2018 2:47 pm

I am looking at SignalGenerator. On Start, the following occurs:
- 60 frames of video are scheduled
- BeginAudioPreroll gets called which initiates RenderAudioSamples at 50K/second
- RenderAudioSamples schedules 1 frame of audio, then it calls StartScheduledPlayback, which immediately ends audio preroll

So, it looks like 60 frames of video and 1 frame of audio is prerolled. Where should I see 3 frames of video and audio being prerolled?

Also, GetBufferedSampleFrameCount is never called. It would be helpful to have an example where all these things are working together to schedule continuous audio/video playback. But, I'll keep trying...
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostMon Jun 25, 2018 3:12 pm

Here is a snippet of what I have so for. It produces sound with an irregular pattern of blank audio.

Code: Select all
void Decklink::Start()
{
  audioChannelCount = 2;
  audioSampleDepth = bmdAudioSampleType16bitInteger;
  audioSampleRate = bmdAudioSampleRate48kHz;
  output->EnableAudioOutput(audioSampleRate, audioSampleDepth, audioChannelCount, bmdAudioOutputStreamContinuous);
  output->SetAudioCallback(this);

  frameWidth = devDisplayMode->GetWidth();
  frameHeight = devDisplayMode->GetHeight();
  devDisplayMode->GetFrameRate(&frameDuration, &frameTimescale);
  framesPerSecond = (float)frameTimescale / (float)frameDuration;

  audioSamplesPerFrame = (unsigned long)((audioSampleRate * frameDuration) / frameTimescale);
  audioBufferSampleLength = (unsigned long)((framesPerSecond * audioSampleRate * frameDuration) / frameTimescale);
  audioBuffer = HeapAlloc(GetProcessHeap(), 0, (audioSamplesPerFrame * audioChannelCount * (audioSampleDepth / 8)));

  totalVideoFramesScheduled = 0;
  for (unsigned long i = 0; i < ringBufferSize; i++)
  {
    output->ScheduleVideoFrame(frame, totalVideoFramesScheduled * frameDuration, frameDuration, frameTimescale);
    totalVideoFramesScheduled++;
  }

  output->BeginAudioPreroll();
}

HRESULT STDMETHODCALLTYPE Decklink::RenderAudioSamples(BOOL preroll)
{
  HRESULT hresult = S_OK;

  unsigned int bufferedSampleCount;
  output->GetBufferedAudioSampleFrameCount(&bufferedSampleCount);

  int sampleSize = audioSamplesPerFrame + (totalAudioSamplesScheduled % 5 == 4 ? 0 : 1);

  if (bufferedSampleCount < sampleSize)
    hresult = output->ScheduleAudioSamples(audioBuffer, sampleSize, 0, 0, NULL);

  totalAudioSamplesScheduled++;

  if (preroll)
    output->StartScheduledPlayback(0, frameTimescale, 1.0);

  return hresult;
}
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostMon Jun 25, 2018 8:17 pm

During preroll, should I schedule all the preroll audio samples at one time?

Code: Select all
// schedule three frames of blank audio preroll
audioBuffer = ONE_FRAME_OF_BLANK_AUDIOSAMPLES;
output->ScheduleAudioSamples(audioBuffer, 801, 0, 0, NULL);
output->ScheduleAudioSamples(audioBuffer, 801, 0, 0, NULL);
output->ScheduleAudioSamples(audioBuffer, 801, 0, 0, NULL);


Or, is this not possible because there is a limit to how many audio samples can be scheduled?
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostWed Jun 27, 2018 7:07 am

Hi Jonas,

Yes, you can schedule all the preroll samples in single RenderAudioSamples callback, then call StartScheduledPlayback. As rule of thumb, you can schedule approx 1 second video + audio, so this should not be a problem.

The issue I see is in your RenderAudioSamples callback itself:
Code: Select all
  if (bufferedSampleCount < sampleSize)
    hresult = output->ScheduleAudioSamples(audioBuffer, sampleSize, 0, 0, NULL);
There are potentially two issues
  1. Your comparison operation (bufferedSampleCount < sampleSize) does not maintain your audio waterlevel set by preroll, it should be (bufferedSampleCount < 3* sampleSize), or similar. Otherwise you will have insufficient audio samples scheduled for embedding in video frame
  2. This conditional statement should be while loop, not if. You audio buffer will be depleted at rate of 59.94Hz (video rate), but RenderAudioSamples callback is called at 50Hz, so there are times you will need to call ScheduleAudioSamples twice in the callback
Both issues can lead to audio buffer becoming depleted which can lead to incomplete samples when scheduling video frame.

Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostFri Jul 13, 2018 4:09 pm

Hi Cameron,

I had to take a break but I'm back at this and I think I'm close. I'm still trying to produce a continuous audio signal by scheduling a frame's worth of samples at a time. Currenlty, the audio has radidly repeating blanks.

A couple of questions...

I am using bmdAudioOutputStreamTimestamped. Earlier you wrote: "streamTime should be an incremental time (0, 801, 1602, ...)." Should the increments always be 801, or should I use an increment of 800 every fifth frame?

I also tried bmdAudioOutputStreamContinuous (and zero's for timeScale and streamTime). In this case, the audio signal is continuous with fewer periodic blanks -- about one blank every 3 or 4 seconds. Eventually, I will want to schedule frames of Windows audio. When I get to this point will that require using bmdAudioOutputStreamTimestamped? I'm just wondering, when is bmdAudioOutputStreamTimestamped necessary?

Thanks,
Jonas
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostFri Jul 13, 2018 8:51 pm

My code currently is as follows.

SETUP...
Code: Select all
   audioBuffer = HeapAlloc(GetProcessHeap(), 0, 3 * (801 * 2 * (16 / 8)));
   INT16* nextBuffer = (INT16*)audioBuffer;
   for (unsigned i = 0; i < 3*801; i++)
   {
      INT16 sample = (INT16)(24576.0 * sin((i * 2.0 * M_PI) / 16));
      for (unsigned ch = 0; ch < 2; ch++)
         *(nextBuffer++) = sample;
   }


   output->EnableAudioOutput(bmdAudioSampleRate48kHz, bmdAudioSampleType16bitInteger, 2, bmdAudioOutputStreamTimestamped);
   output->SetAudioCallback(this);
   output->BeginAudioPreroll();


Code: Select all
HRESULT STDMETHODCALLTYPE Decklink::RenderAudioSamples(BOOL preroll)
{
   const unsigned int minBufferSize = 48000/50;
   unsigned int bufferedSampleCount = 0;
   output->GetBufferedAudioSampleFrameCount(&bufferedSampleCount);
   unsigned int samplesToWrite = 0;

   if (preroll)
   {
      while (totalAudioSamplesScheduled < 3)
      {
         unsigned long currentSampleSize = (totalAudioSamplesScheduled++ % 5 == 4 ? 800 : 801);
         samplesToWrite += currentSampleSize;
      }
      output->StartScheduledPlayback(0, frameTimescale, 1.0);
   }
   else
   {
      unsigned int samplesUnderflowCount = max(0, (int)minBufferSize - (int)bufferedSampleCount);
      while (samplesToWrite < samplesUnderflowCount)
      {
         unsigned long currentSampleSize = (totalAudioSamplesScheduled++ % 5 == 4 ? 800 : 801);
         samplesToWrite += currentSampleSize;
      }
   }

   if (samplesToWrite == 0)
      return S_OK;
   else
      return output->ScheduleAudioSamples(audioBuffer, samplesToWrite, totalAudioSamplesScheduled*801, 48000, NULL);
}
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: ScheduleAudioSamples

PostTue Jul 17, 2018 6:35 am

Hi Jonas,

I did notice 2 potential issues:
1) In your RenderAudioSamples callback, ensure that your call to ScheduleAudioSamples is before call to StartScheduledPlayback, otherwise your playback will begin without scheduled audio samples.

2) It seems that the streamTime that you are writing to ScheduleAudioSamples represents the stream time at end of audio buffer, not beginning. Also you should maintain a streamTime that increments by samplesToWrite at the end of RenderAudioSamples callback.

Kind Regards
Cameron
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostMon Jul 23, 2018 8:23 pm

Two questions.

1) Should streamTime increment by 801 always?
0, 801, 1602, 2403, 3204, 4005, 4806,...
Or, should it take into account the actual rate of 800.8?
0, 801, 1602, 2403, 3204, 4004, 4805,...

2) If I ever can get this constant signal working, next I will want to get audio samples from Windows synced to video. For this case, is it required that I use bmdAudioOutputStreamTimestamped?
Offline

Jonas Geduldig

  • Posts: 39
  • Joined: Tue Jun 17, 2014 7:04 pm

Re: ScheduleAudioSamples

PostThu Jul 26, 2018 5:10 pm

Finally got it working for playing a constant pitch while scheduling audio samples per each frame.

Setup...
Code: Select all
   prerollFrames = 5;
   cyclesPerPreroll = 48;
   audioChannelCount = 2;
   audioSampleDepth = bmdAudioSampleType16bitInteger;
   audioSampleRate = bmdAudioSampleRate48kHz;

   double audioSamplesPerFrame = (double)(audioSampleRate * frameDuration) / (double)frameTimescale;
   audioSamplesPerFrameFloor = floor(audioSamplesPerFrame);
   audioSamplesPerFrameCeil = ceil(audioSamplesPerFrame);

   audioPrerollSampleSize = 0;
   for (int i = 0; i < prerollFrames; i++)
   {
      unsigned int sampleSize = (totalAudioSamplesScheduled++ % 5 == 4 ? audioSamplesPerFrameFloor : audioSamplesPerFrameCeil);
      audioPrerollSampleSize += sampleSize;
   }

   audioBuffer = HeapAlloc(GetProcessHeap(), 0, audioPrerollSampleSize * audioChannelCount * audioSampleDepth / 8);
   double samplesPerCycle = audioPrerollSampleSize / cyclesPerPreroll;
   INT16* nextBuffer = (INT16*)audioBuffer;
   for (unsigned int i = 0; i < prerollFrames*audioSamplesPerFrameCeil; i++)
   {
      INT16 sample = (INT16)(24576.0 * sin((i * 2.0 * M_PI) / samplesPerCycle));
      for (unsigned ch = 0; ch < channels; ch++)
         *(nextBuffer++) = sample;
   }

   output->EnableAudioOutput(audioSampleRate, audioSampleDepth, audioChannelCount, bmdAudioOutputStreamTimestamped);
   output->SetAudioCallback(this);
   output->BeginAudioPreroll();


Callback to buffer more audio samples...
Code: Select all
HRESULT STDMETHODCALLTYPE Decklink::RenderAudioSamples(BOOL preroll)
{
   HRESULT hresult = S_OK;
   unsigned int requiredAudioSamples;
   unsigned int samplesToWrite = 0;

   if (preroll)
      requiredAudioSamples = audioPrerollSampleSize;
   else
   {
      unsigned int bufferedSampleCount = 0;
      hresult = output->GetBufferedAudioSampleFrameCount(&bufferedSampleCount);
      if (hresult != S_OK)
         return hresult;
      unsigned int minBufferSize = audioSampleRate / 50;
      requiredAudioSamples = max(0, (int)minBufferSize - (int)bufferedSampleCount);
   }

   while (samplesToWrite < requiredAudioSamples)
   {
      unsigned int sampleSize = (totalAudioSamplesScheduled++ % 5 == 4 ? audioSamplesPerFrameFloor : audioSamplesPerFrameCeil);
      samplesToWrite += sampleSize;
   }

   if (samplesToWrite > 0)
   {
      double samplesPerCycle = audioPrerollSampleSize / cyclesPerPreroll;
      samplesToWrite = ceil(double(samplesToWrite) / samplesPerCycle) * samplesPerCycle;
      hresult = output->ScheduleAudioSamples(audioBuffer, samplesToWrite, audioStreamTime, audioSampleRate, NULL);

      audioStreamTime += samplesToWrite;

      if (hresult == S_OK && preroll)
         hresult = output->StartScheduledPlayback(0, frameTimescale, 1.0);
   }

   return hresult;
}

Return to Software Developers

Who is online

Users browsing this forum: No registered users and 11 guests