GStreamer Pipeline for Live Audio and Video

Ask software engineering and SDK questions for developers working on Mac OS X, Windows or Linux.
  • Author
  • Message
Offline

Gareth Halfacree

  • Posts: 8
  • Joined: Mon Jan 29, 2018 2:51 pm
  • Real Name: Gareth Halfacree

GStreamer Pipeline for Live Audio and Video

PostMon Jan 29, 2018 2:58 pm

Apologies if this isn't the right subforum, but it looks like my best shot.

I've got a Blackmagic Intensity Pro (non-4K), which I primarily use for grabbing audio and video from devices under review. I also use it so I can view the video output of devices during the test without needing a second monitor, using a simple GStreamer pipeline to show live video:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=5 ! videoconvert ! xvimagesink

That works a treat, but I'd now like to include live audio - and that's where I've come unstuck. I know there's a "decklinkaudiosrc" which does for audio what "decklinkvideosrc" does for video, but every attempt I've made at including it in the pipeline - with various "queue", "mux," and "tee name=t" commands interspersed plus a "pulsesink" at the end so I can actually get the audio to play - fails miserably, typically by complaining that GStreamer can't link to "decklinkvideosrc."

I've found examples for capturing video and audio to a file using GStreamer, but not for live display - and, again, have failed miserably in attempting to modify them for same.

So, could any kind person share a working GStreamer pipeline that will display live video in an xvimagesink and play live audio through pulsesink (or any other audio sink compatible with Ubuntu 16.04.3)?

Thanks in advance!
Offline

Gareth Halfacree

  • Posts: 8
  • Joined: Mon Jan 29, 2018 2:51 pm
  • Real Name: Gareth Halfacree

Re: GStreamer Pipeline for Live Audio and Video

PostMon Jan 29, 2018 10:18 pm

Update:
The following gives me roughly 0.2 seconds of choppy audio every 1.5 seconds:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=5 ! videoconvert ! xvimagesink decklinkaudiosrc typefind=true do-timestamp=true alignment-threshold=100 ! audioconvert ! autoaudiosink

Turning off the audio sync turns that into continuous audio but really, really choppy - so much so that if the source isn't actually playing audio there's a continuous popping hum:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=5 ! videoconvert ! xvimagesink decklinkaudiosrc typefind=true do-timestamp=true alignment-threshold=100 ! audioconvert ! autoaudiosink sync=false

No amount of adding queues in there changes anything.

Any ideas?
Offline

Dave Johnstone

  • Posts: 223
  • Joined: Mon Dec 08, 2014 11:12 am
  • Location: Adelaide

Re: GStreamer Pipeline for Live Audio and Video

PostTue Jan 30, 2018 12:56 am

My system will play video and audio in sync with the following:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=3 ! autovideoconvert ! xvimagesink decklinkaudiosrc ! audioconvert ! autoaudiosink

This is with an HD Extreme 3D+ card on Debian 9 64-bit being fed SDSDI from a Digibeta source.

Regards,

Dave.
Freelance Television Broadcast Engineer
and Software Programmer.
Offline

Gareth Halfacree

  • Posts: 8
  • Joined: Mon Jan 29, 2018 2:51 pm
  • Real Name: Gareth Halfacree

Re: GStreamer Pipeline for Live Audio and Video

PostTue Jan 30, 2018 9:38 am

Dave Johnstone wrote:My system will play video and audio in sync with the following:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=3 ! autovideoconvert ! xvimagesink decklinkaudiosrc ! audioconvert ! autoaudiosink

This is with an HD Extreme 3D+ card on Debian 9 64-bit being fed SDSDI from a Digibeta source.

Sadly, that just gives me the same effect as my original pipeline: a pop and a fraction of a second of audio every 1.5 seconds, otherwise silence.

I'm running GStreamer 1.8.3 on Ubuntu 16.04.3, using the latest packages and kernel available from the Canonical repos.
Offline

Dave Johnstone

  • Posts: 223
  • Joined: Mon Dec 08, 2014 11:12 am
  • Location: Adelaide

Re: GStreamer Pipeline for Live Audio and Video

PostTue Jan 30, 2018 9:41 pm

I have Ubuntu 15.10 installed on another partition in the same system so I tested with that. Same result: video and audio working correctly in sync. GStreamer on this system is 1.6.3.

I forgot to mention the simplest test in my previous post - have you tested in Media Express? I assume you'd see the audio meters dropping unexpectedly or hear the problem in a capture file.

Regards,

Dave.
Freelance Television Broadcast Engineer
and Software Programmer.
Offline

Gareth Halfacree

  • Posts: 8
  • Joined: Mon Jan 29, 2018 2:51 pm
  • Real Name: Gareth Halfacree

Re: GStreamer Pipeline for Live Audio and Video

PostTue Jan 30, 2018 9:48 pm

Dave Johnstone wrote:I have Ubuntu 15.10 installed on another partition in the same system so I tested with that. Same result: video and audio working correctly in sync. GStreamer on this system is 1.6.3.
Strange - there's nothing I can do on my system to make it work.
Dave Johnstone wrote:I forgot to mention the simplest test in my previous post - have you tested in Media Express? I assume you'd see the audio meters dropping unexpectedly or hear the problem in a capture file.
It's absolutely fine through Media Express (except, of course, that I can't actually hear anything until I play back the recording) - so there's nothing wrong with the hardware.

Most odd!
Offline

Dave Johnstone

  • Posts: 223
  • Joined: Mon Dec 08, 2014 11:12 am
  • Location: Adelaide

Re: GStreamer Pipeline for Live Audio and Video

PostWed Jan 31, 2018 5:38 am

You could try the GStreamer debug log to see if it indicates any problem. Enter
Code: Select all
export GST_DEBUG=decklinkaudiosrc:5
Then run the pipeline again. This is a snippet of my output:
Code: Select all
gst-launch-1.0 decklinkvideosrc mode=3 ! autovideoconvert ! xvimagesink decklinkaudiosrc ! audioconvert ! autoaudiosink
Setting pipeline to PAUSED ...
0:00:00.037165003  2435 0x56510c514760 DEBUG       decklinkaudiosrc gstdecklinkaudiosrc.cpp:675:gst_decklink_audio_src_open:<decklinkaudiosrc0> Opening
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
0:00:00.135975945  2435 0x56510c4dd1e0 DEBUG       decklinkaudiosrc gstdecklinkaudiosrc.cpp:278:gst_decklink_audio_src_set_caps:<decklinkaudiosrc0> Setting caps audio/x-raw, rate=(int)48000, layout=(string)interleaved, format=(string)S16LE, channels=(int)2
New clock: GstDecklinkInputClock
Redistribute latency...
0:00:00.225362824  2435 0x56510c4dd1e0 DEBUG       decklinkaudiosrc gstdecklinkaudiosrc.cpp:600:gst_decklink_audio_src_create:<decklinkaudiosrc0> Outputting buffer 0x56510c504cc0 with timestamp 0:00:00.048515097 and duration 0:00:00.039984903
0:00:00.227282412  2435 0x56510c4dd1e0 DEBUG       decklinkaudiosrc gstdecklinkaudiosrc.cpp:600:gst_decklink_audio_src_create:<decklinkaudiosrc0> Outputting buffer 0x7f4ed4017060 with timestamp 0:00:00.088500000 and duration 0:00:00.040000000
Regards,

Dave.
Freelance Television Broadcast Engineer
and Software Programmer.

Return to Software Developers

Who is online

Users browsing this forum: roger.magnusson and 19 guests