Audio+Video capture in C#

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

Nono3551

  • Posts: 5
  • Joined: Fri Aug 13, 2021 2:15 pm
  • Real Name: Michal Zahradnik

Audio+Video capture in C#

PostThu Feb 24, 2022 9:57 am

Hi, I am new to decklink SDK and I am trying to capture&preview video+audio from camera using UltraStudio HD Mini.

I made video preview working but I had issues with paying sound from camera in real time. Sound is out of sync and faster than real sound. I noticed this when I buffered multiple packets of sound and played them together.

I am capturing audio and video in IDeckLinkInputCallback.VideoInputFrameArrived. For audio playback I am using NAudio.

Code: Select all

        public uint channelCount { get; set; } = 2;
        public _BMDAudioSampleType sampleType { get; set; } = _BMDAudioSampleType.bmdAudioSampleType16bitInteger;
        public int sampleTypeInt { get; set; } = 16;
        private WaveFormat waveFormatTarget = new WaveFormat(48000, sampleTypeInt, (int) channelCount);
        private BufferedWaveProvider waveProvider;
        private WaveOut waveOut = new();
        private Queue<byte[]> audioSamplesQueue = new();


Code: Select all
            waveOut.Init(waveProvider);
            waveOut.Play();
            waveOut.Volume = 1;


Code: Select all
private void BufferThread()
        {
            Task.Run(() =>
            {
                while (true)
                {
                    Thread.Sleep(10000);
                    try
                    {
                        while (audioSamplesQueue.Count > 0)
                        {
                            var buffer = audioSamplesQueue.Dequeue();

                            waveProvider.AddSamples(buffer, 0, buffer.Length);

                            memoryStream.Write(buffer);
                            memoryStream.Flush();
                        }
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(e);
                    }
                }
            });
        }


Code: Select all
            if (audioPacket != null)
            {
                audioPacket.GetBytes(out var inputAudioBuffer);

                if (inputAudioBuffer != IntPtr.Zero)
                {
                    var sampleCount = audioPacket.GetSampleFrameCount();
                    var bufferLength = waveFormatTarget.BitsPerSample / 8 * waveFormatTarget.Channels * sampleCount;
                    var tempBuffer = new byte[bufferLength];

                    audioPacket.GetPacketTime(out var packetTime, 10 * 1000 * 1000);

                    Marshal.Copy(inputAudioBuffer, tempBuffer, 0, bufferLength);

                    Marshal.ReleaseComObject(audioPacket);

                    audioSamplesQueue.Enqueue(tempBuffer);
                }
                else
                {
                    Console.WriteLine();
                }
            }
Offline

Cameron Nichols

Blackmagic Design

  • Posts: 442
  • Joined: Mon Sep 04, 2017 4:05 am

Re: Audio+Video capture in C#

PostTue Mar 01, 2022 6:30 am

Hi Michal,

Can you instead try holding onto the IDeckLinkAudioInputPacket reference and store this in a queue. Later you can call Marshal.Copy/Queue.Enqueue in your processing thread.

My concern is that you should keep IDeckLinkInputCallback::VideoInputFrameArrived as lightweight as possible and that these methods will be relatively processor intensive with copy and reallocations.

You can test this theory by keeping log of the difference in the timestamps from IDeckLinkInput::GetHardwareReferenceClock called at the start of callback with IDeckLinkVideoInputFrame::GetHardwareReferenceTimestamp. This will provide the time for callback to occur as compared to the End of Frame on the wire.

If you see this difference grow, then it would suggest the callback is not keeping up with the frame rate. This may lead to later dropping audio samples as the input buffer fills.

Regards
Cameron
Offline

Nono3551

  • Posts: 5
  • Joined: Fri Aug 13, 2021 2:15 pm
  • Real Name: Michal Zahradnik

Re: Audio+Video capture in C#

PostTue May 17, 2022 8:47 am

I made example with winui 3 preview. It just work (for a while). Feel free to inspire. Repository here:
https://github.com/nono3551/BM-Capture

Return to Software Developers

Who is online

Users browsing this forum: No registered users and 20 guests