Changing stream dimensions in .NET, C#, WPF, MediaFoundation

I'm having some trouble with programmatically setting up my encoding profile for my ATEM Television Studio and I was hoping someone might have some advice.

I am working with the ATEM Switcher 6.6.1 SDK (which appears to include the Desktop Video 10.5.0 SDK). The application is written in C# using .NET, WPF, and Windows Media Foundation.

Some Background:

For no particular reason other than it seemed like a reasonable starting point, I have been using the "AppleTV" profile as my IBMDStreamingVideoEncodingMode and then calling CreateMutableVideoEncodingMode() to create a IBMDStreamingMutableVideoEncodingMode. Once I have a mutable encoding mode, I am setting the destination size using SetDestSize() to 1280x720 and begin receiving packets. As packets arrive I feed them into the media foundation H264 decoder. After the H264 decoder has enough information to make sense out of the packets it is decoding, it will trigger a MF_E_TRANSFORM_STREAM_CHANGE to alert me that my output stream needs to be updated to match the incoming data. At this point, my code can enumerate the available output types on the decoder using GetOutputAvailableType() and select the 'best' output type. The interesting thing here is that if I query the available output types, they all report a size of 1280x720, which matches what I expected.

The Problem:

In an effort to improve the quality of my encoded video, I decided to try upping my settings to capture in 1920x1080. To do this, I changed my starting profile to "Native", created a mutable encoding mode, and set the destination size using SetDestSize() to 1920x1080. Oddly however, when I receive the MF_E_TRANSFORM_STREAM_CHANGE notification, all of the available output types still report a size of 1280x720.

I could not find any way to effect this until a lucky accident.

I tried performing an encode using the ATEM Switcher software and the "Native" profile; which worked fine. When I returned to my code, I noticed the available output types all now report 1920x1088. (Not, it's not a typo, they really report 1920x1088). On a lark, I tried using the ATEM software to encode using the "AppleTV" profile and then exercising my code reports 1280x720.

My conclusion:

The best guess I have right now is the SetDestSize() method of the mutable profile doesn't do exactly what I think is does. And changing the profile does not seem to impact the dimensions of the stream I receive. There must be some other way to affect the dimensions of the stream I am receiving because clearly changing the profile in the ATEM software *will* impact the values I receive in Media Foundation during the MF_E_TRANSFORM_STREAM_CHANGE notification (which are derived from the packets I have been feeding into the decoder). But so far, I can't seem to find any method of affecting the stream dimensions programmatically.

On a related note; why would a 1920x1080 profile report 1920x1088? But the 1280x720 profiles report 1280x720 as expected?


Can anyone shed any light on this for me? It makes sense that I should be able to know what dimensions to expect from the ATEM, and that I should be able to set this programmatically.

Your question is confusing, as it almost sounds like you are trying to control the output of the H.264 encoder separately from the ATEM Video mode. Or you are intending to decode and re-encode the data from the H.264 encoder within software on your computer - Big overhead!!

If you are using a TVS, it basically has a BMD H.264 encoder within it, so although you are using the Decklink SDK, it's like a separate API and shares very few of the functions / techniques. I'd be surprised if the H.264 resampled frame sizes, but I do like it when BMD do things that surprise me, it's a while since I last delved into the H.264 encoder, but from memory; the H.264 NAL packets always contained data that matched the video being fed into it, so if you want a 1280x720 stream, the ATEM would need to be in 720p mode. for 720x576, you need the ATEM in PAL mode, and for 1920x1080 a 1080i/p mode.

The encoded size 1920x1088 issue you are reporting is an H.264 thing. Vertical resolution is always a multiple of 16 pixels, due to the way H.264 works, All H.264 works this way, and the MUX headers normally hide this fact from the end user. Those lines should be black and anything rendering that stream should discard the last 8 lines.

Depending on where you are sending a H.264 stream determines the presets and profile and Level you should use. My advice if using MFC to MUX your stream out to file, is to start out with Main or High (maybe Baseline for low-quality) for AVC preset / profile. The level you select is determined by the a number of settings, and if you try incorrect combinations of Profile and Level, you can get encoder failures.

Although you don't really mention what it is you are doing: My guess is you want to do something with the uncompressed frames like show them on a screen, in which case the H.264 USB2 output of the TVS is not the best method. You should use an SDI / HDMI output and capture using a Decklink uncompressed video processing card. The USB2 output is good for sending to a streaming platform (e.g. Youtube or UStream or any of the many others), or for logging your output to disk, but you are introducing a significant processing overhead and several seconds of latency for every other use-case.

Essentially this post is confusing because you are using the TVS's ability to encode H264, only to decode it again at the end of a 2m USB cable... I can't think of any reasons why would you would want do that over simply ingesting the uncompressed frames as you'll find the uncompressed frames a h3ll of a lot easier to deal with?

John,

Thank you for the detailed response. I appreciate your taking some time to try and help me. I apologize for the length of this post, hopefully it will help make my question more clear.

By way of background, I have several Marshall 1080 HD-SDI cameras mounted at a small (local) sporting venue. We feed the various cameras into a ATEM Televison Studio, and allow our software to select the appropriate camera inputs, upload stills graphics (scores, leaderboards, etc.) and overlay the upstream key to produce 'live at the event' program output.

We also want to capture this output to disk for later use. My strategy to accomplish this was to use the DeckLink SDK to capture the H264 video NAL and audio packets, decode them and allow windows media foundation to encode an output stream (wmv, mp4, or whatever).

I recognize there may be some performance issues related to decompressing and re-compressing the streaming packets. If I could figure out how to feed the received H264 packets *directly* into media foundation, that would certainly be a better solution, but for now it is adequate to decode and re-encode my output. (Yes, I recognize there are alternative capture solutions available, but the design specification I have to work within limits me to using the USB H264 packets).

I know the ATEM is set to 1080i 59.94 because this is the ONLY setting that works with my specific cameras. If the ATEM changed modes, the inputs would stop working. That is NOT happening, so I am reasonably sure that the ATEM is set to 1080i 59.94 and not changing. I also query and include this as part of my trace log to ensure the input mode is correct.

Unfortunately for me, I am trying to understand something that seems to live at the intersection between the ATEM SDK and Windows Media Foundation; and I am not sure which side of the fence to pursue.


Consider the following two captures from the trace log produced by my software:

EXAMPLE 1:

Preset: YouTube 720p (5)

Current input mode: 'HD1080i5994'
Decklink Source: (1920x1080) @ (0,0)
Decklink Destination: (1280x720)
Audio Bitrate: 160 Kbps
Audio Channels: 2
Audio Codec: 'AAC'
Audio Sample Rate: 48000
H264 EntropyCoding: 'CABAC'
H264 Has BFrames: True
H264 Level: 'Level4'
H264 Profile: 'High'
Video Bitrate: 8000 Kbps
Video Framerate: 'Framerate_2997p'

I420: 1280x720
IYUV: 1280x720
NV12: 1280x720
YUY2: 1280x720
YV12: 1280x720

EXAMPLE 2:

Preset: YouTube 720p (5)

Current input mode: 'HD1080i5994'
Decklink Source: (1920x1080) @ (0,0)
Decklink Destination: (1280x720)
Audio Bitrate: 160 Kbps
Audio Channels: 2
Audio Codec: 'AAC'
Audio Sample Rate: 48000
H264 EntropyCoding: 'CABAC'
H264 Has BFrames: True
H264 Level: 'Level4'
H264 Profile: 'High'
Video Bitrate: 8000 Kbps
Video Framerate: 'Framerate_2997p'

I420: 1920x1088
IYUV: 1920x1088
NV12: 1920x1088
YUY2: 1920x1088
YV12: 1920x1088

This output was generated using the built in 'YouTube 720p' encoding profile as is. I did not create a mutable encoding profile. In both examples I am executing exactly the same code. Yet in the first example, when Media Foundation has ingested enough H264 packets to make a recommendation for the decoder output format (I420, IYUV, NV12, YUY2, or YV12), it recommends 1280x720 for all available outputs.

The same test can also produce 1920x1088 as a recommended output.

What I can not seem to figure out is HOW this changes? I can not discover any programmatic way to set this or predict which recommendation I will receive from Windows Media Foundation. What I do know is that the recommendation should (theoretically) be based on the packets being ingested by the decoder, which leads me back to the ATEM and the DeckLink SDK.

I admit I am quite perplexed at this point. I've done dozens of tests and I can not find a reliable or predictable pattern to 'switching' this value. I have had some limited success cycling my ATEM, and changing profiles using the ATEM Software Control, but neither seem to be a reliable or predictable way to cause this value to change.

Just a quick note: It *appears* that cycling my ATEM (leaving it off for a good 30 seconds) will cause my code to return to always recommending 1280x720. I haven't figured out how to 'trigger' it back into the 1920x1088 except I know it somehow involves using the ATEM Software Control.

More interestingly, if I do get my ATEM/Media Foundation to recommend I use 1920x1088, the Media Foundation H264 decoder will fail to output data. It will ingest several packets (ie. NEED_MORE_INPUT) and eventually it will return "UNSPECIFIED_ERROR" trying to output.

I wonder if the ATEM is *supposed* to only output 720p on the USB stream, and the 1920x1088 is actually some kind of error condition? I looked at the technical specs for the ATEM, but I couldn't find any definite specifications for the H264 USB stream.

I know it seems like the ATEM is producing the H.264, and whereas this is true in in that H.264 encoder functionality is within the ATEM, it's best to consider it a separate device downstream of the ATEM. They don't interact in terms of control, and just as with a separate H.264, you shouldn't attempt to change the Video mode being fed into it, or the Encoding parameters (EncodeMode) whilst it is actually streaming, and hence my first bit of advice is to ensure you stop the streaming interface, then change video mode, then restart your stream.

(*) IBMDStreamingDeviceInput.StopCapture

Then change the ATEM Video Format

(*) IBMDSwitcher.SetVideoMode
or
(*) IBMDSwitcher_v4_0.SetInt( bmdSwitcherPropertyIdVideoMode_v4_0, [MODE] )

Wait for the H.264 to report it is seeing the video mode that matches matches the ATEM.

(*) IStreamingDeviceInput.GetCurrentDetectedVideoInputMode

Then set the Input mode on the H.264 for good measure

(*) IStreamingDeviceInput.SetVideoInputMode

Finally set the encoding mode

(*) IStreamingDeviceInput.SetVideoEncodingMode

Then restart the capture

(*) IStreamingDeviceInput.StartCapture

Worth Checking that the current encoding mode supports the Video Input mode, and visa versa with:

(*) IStreamingDeviceInput.DoesSupportVideoInputMode
(*) IStreamingDeviceInput.DoesSupportVideoEncodingMode

What are you getting back from "IStreamingDeviceInput.DoesSupportVideoEncodingMode"?

The ATEMs are not unstable in any way, and do not need rebooting, my guess is you are actually resetting the H.264 encoder when you power cycle the ATEM. More specifically you are stopping the device from streaming and hence freeing it up to start encoding in a new mode again.

So having re-read all these posts a few times and looked again at the H.264 SDK (albeit a slightly out-of-date version), I think your problems boil down to the basic sequence of calls to enable and disable the stream when setting / changing modes.

If that doesn't solve it, try writing the NAL / Audio packet data to disc as raw data files in the different scenarios of it working and when it doesn't work, it should be fairly obvious if the encoder has stopped outputting data. I'm 100% certain the H.264 encoder will output a 1080 H.264 stream, I'd be utterly astonished if the H.264 encoder within the TVS was different.

John,

> "The ATEMs are not unstable in any way, and do not need rebooting, my guess is you are actually resetting the H.264 encoder when you power cycle the ATEM. More specifically you are stopping the device from streaming and hence freeing it up to start encoding in a new mode again."

Agreed. I didn't mean to imply the switcher was unstable; merely that the only thing I could seeming do to impact my issue was to reset it.

I liked your train of thought about the order of operation, so I double (triple) checked my code. If I am getting out of sequence, I sure can't find it, but I will continue to keep looking at it.

Also, I liked your logging idea and I am heading in that direction next.

Your help and input is appreciated. Thanks!