INTRODUCTION/STATEMENT OF PROBLEMThis is my report and analysis of the ATEM Mini Pro's "Cache Full" problem. While streaming, the streaming status area should display OK to indicate 0% cache in use. Users have experienced circumstances where the cache % will appear and start to increase, reach 100% (at which point the cache status will display "FULL"), leading to "failure" of the stream. Making matters more stressful, pressing the OFF button to attempt to end the stream (so as to restart it), results in no feedback that the OFF button has been pressed, and the stream appears to be locked in the On Air position but with the warning that 100% of the cache is full. The operator feels that cutting power to the ATEM is the only way to regain control of the ATEM (it isn't, but I understand the panic and belief that this is the only way to regain control).
I experienced this twice on 9/19/2020 while streaming, once just prior to the start of a wedding ceremony, then again in the middle of the ceremony. After the first occurrence, I tried to stop the stream and ended up pulling the power cord as it appeared that the ATEM had "locked up" (it hadn't - more on that later). I reduced the outgoing bitrate from 7Mbits/sec to 4.5Mbits/sec (bandwidth had been measured prior to the event at 11Mbits/sec and I was connected via ethernet cable directly into the router). Because the second occurrence happened in the middle of the ceremony, cutting power to the ATEM would have been disastrous, so I relied on the fact that I was simultaneously recording using the ATEM's built-in record-to-flash-drive feature. Although this "saved the day" (I uploaded the recording during the reception so by the time I left the event, the "playback" was live), I wasn't sure what experience the remote audience had during the second incident. I ended up switching to OBS for the reception just to be safe. Anecdotally but importantly, another vendor stated that at the same time I experienced the second incident that he appeared to lose control of his internet-controlled smart lighting of the event. He felt that the network was offline for some period of time before he was "back online" and able to control his internet-controlled lighting once again.
This "cache full" issue has been discussed on the manufacturer's forum for months (including the thread where I am posting this) and has been a problem with many streaming hosts (I am streaming to Mixcloud but the problem has been experienced with every streaming platform).
Proposed solutions by users have included all of the following, none of which solve the problem:
- disconnect the control cable (USB)
- increase the outgoing streaming bitrate
- put the ATEM on a laptop cooler (user thought it solved the problem, then retracted as a fluke)
- increase the outgoing audio bitrate
EXECUTIVE SUMMARYIn Sept 2019, OBS v24.0 introduced the advanced option to stream using a "dynamically variable upstream bitrate". This resolved many issues related to experiencing "dropped frames" due to unpredictable network congestion. Dynamic Bitrate was a game changer. Now, all stuttering/pausing due to dropped frames was a thing of the past. Instead, OBS will ratchet down the quality (intentionally) during the timeframe when the upstream bandwidth is hampered, then resume the intended bitrate when the network is back to expected bandwidth. The idea is that seeing extreme artifacting is better than experiencing dropped frames. I agree.
By contrast, the ATEM Mini Pro handles poor upstream bandwidth by use of an onboard memory cache. If upstream bandwidth becomes constrained and the intended data cannot be sent, the data is re-routed to a modest-sized memory cache within the ATEM hardware. This cache is a FIFO (First In First Out); as the network "clears up", the backlogged data is sent out. While the constrained bandwidth is occurring, the remote audience sees dropped frames and freezing (pausing) of the video. Note that audio is prioritized as much as possible, so often the remote audience will hear segments of uninterrupted audio with a very low frame rate (extremely low sometimes, like 1 frame onscreen for several seconds, then another frame, and so on). Generally, the remote audience will continue to see an extremely poorly performing stream (as opposed to complete cessation). To the operator, however, it will feel like the stream has completely stopped (it hasn't but the quality of the stream is so poor as to be not much better than completely stopped).
After 6 consecutive hours of extensive testing of a wide variety of variables and observation platforms, I have characterized the problem and have devised methodologies to mitigate it, and to minimize the effect of the problem on the remote audience. Skip down to "solution" if you don't care about the testing and analysis.
TESTING AND ANALYSISThe first step was to recreate the Cache Full problem. This proved to be harder than it sounds! Even with the ATEM set to 9Mbits/sec, I was unable to experience the Cache Full problem. My test network supports 11Mbits/sec upstream bandwidth quite reliably, so the ATEM happily streamed at 9Mbits/sec. To create the problem, then, I set several other computers on the same network to upload 1GB dummy files to offsite servers, simultaneously while streaming. This ended up being a highly reliable way to create the problem. I could now experience the Cache Full problem "upon demand" and thus run experiments.
First, to understand the remote audience experience during these times, I used my cell phone on a completely separate network (LTE cellular data) to observe the stream. I saw a stream with poor performance - lots of frozen frames and pausing of the video. I was surprised to observe that for long stretches of time, the audio played smoothly even with an extremely low video frame rate. The ATEM appears to prioritize sending audio since it requires much less bandwidth. The stream was effectively unwatchable, however, due to the erratic frame rate and long moments of frozen frames.
What happens when the OFF button is pressed on the ATEM to stop the stream? As many have observed, it appears that nothing happens. There is no visual indicator that the OFF button has been pressed, and the operator appears to lack control of the ATEM hardware or the software controller. In other words, if the intent is to stop the stream, reduce the streaming bitrate, then start the stream, this appears to be impossible to execute because the ATEM (hardware and software control) appears completely unresponsive.
Not so fast. It turns out that this is an exercise in patience. When the OFF button is pressed to stop streaming, the ATEM faithfully stops attempting to fill the cache. In effect, anything happening "live" from the moment you press the OFF button CEASES to be captured by the ATEM. (Note that if using the Record function to record to a connected flash drive, the recording does continue, unaffected by all of this.) But the reason the OFF button doesn't appear to work ("ON AIR" continues to be lit no matter how many times you press OFF) is because the ATEM is faithfully continuing to pursue streaming of the content stored within the cache. If you wait long enough, you'll see the cache indicator drop from FULL to 99% then gradually work its way down all the way to 0%. Once the cache has emptied out to the stream, the OFF button lights up and the operator once again has control. You can change the streaming bitrate at this point, and you can press the ON AIR button again. Because the emptying of the cache can take a long time (in many cases it took 2 minutes during my testing) most operators will "give up" before that length of time, thinking the unit has become completely unresponsive and simply pull the power and reboot the unit. This it not advised as the remote audience gets a very abrupt halt of the stream.
So what happens if you wait out the emptying of the cache, reduce the bitrate, then go ON AIR again? I'll illustrate with a sample timeline:
1:00:00 cache starts to fill up
1:00:30 cache hits full within 30 seconds; operator presses OFF on ATEM (or in software)
1:00:31 no more video enters the cache, cache starts to empty out, but ATEM still shows "ON AIR"
.
.
.
1:01:30 a full minute later, the cache has emptied out, OFF button finally lights up
1:01:31 operator reduces bitrate, then presses ON AIR again
1:01:32 stream resumes (at the now lower bitrate)
From the remote audience viewpoint, this is what they saw:
1:00:00 stream starts to freeze and pause
...it takes a minute and a half to play the 30 seconds of live footage from 1:00:00 to 1:00:30...
1:01:32 stream resumes, but the footage from 1:00:30 through 1:01:31 was never seen
Clearly, 60 seconds of missing live footage is quite undesirable. The remote audience endures 90 seconds of a very poor quality stream that only covers 30 seconds of live footage, then when things resolve, we've "jumped forward" a minute, and the missing 60 seconds is never seen.
SOLUTIONThe natural tendency of the operator when observing the cache filling up, is to hope that the % number will go back down and the problem will resolve itself. Don't wait. If the cache gets above 40%, press OFF immediately. I've found that it takes much less time to empty the cache if you do so at 40% than if you wait until it's FULL. You'll regain control of the ATEM within mere seconds as opposed to waiting a full minute to regain control. To clarify, as soon as that cache gets above 40%, press OFF, wait a few seconds and the OFF light will light up acknowledging the stream has been stopped and the cache has emptied, then lower your outgoing bitrate, then press ON AIR and now you're streaming again and your remote audience has only missed the few seconds of live footage while you were waiting for the ATEM to empty the small amount in the cache. I tested this repeatedly both ways, and emptying out a 40% cache is reasonably quick, as opposed to emptying a FULL cache. Remember, since the network is apparently unable to keep up with the bitrate, a FULL cache is not just two and a half times a 40% cache. It's much worse because it's having trouble streaming anyway, and now it has to endure that trouble for 2.5 times the amount of DATA which itself was uploading at a slower rate so will take much longer than 2.5 times the amount of TIME. Think of it like a clogged bathtub drain - if you detect the clog early, you can shut off the water supply and the small amount of water in the bathtub will drain reasonably fast through the clogged drain. But if you let the bathtub fill completely with water, it's going to be a very long wait for that huge volume of water to drain through the clogged drain. Similarly, it's easy to fill the cache, but much harder for the cache to empty through a clogged network connection.
Definitely have a flash drive plugged into the ATEM and be recording simultaneously so that you have a recording of the footage that ended up missing from the livestream.
Is this a perfect solution? Of course not. But it does minimize the problem to the remote audience, and it does minimize the amount of "missing footage" not seen by the remote audience.
RECOMMENDATION TO BLACKMAGIC DESIGNRecognize that the solution given above is a "poor man's" version of what OBS is doing with its Dynamic Bitrate option. You stop the stream (as soon as the problem starts happening), reduce the bitrate, then go back on air. BlackMagic Design could fix this problem by implementing a Dynamic Bitrate option similar to the one in OBS. It's not difficult for BlackMagic Design to do so, because they're ALREADY detecting that an overflow condition is occurring because they're filling the onboard cache and reporting to the operator that they're doing so! So instead of filling up the cache, why not just reduce the bitrate within the software, monitor the buffer overflow condition, and if it's still backlogged reduce the bitrate further, and so on. The remote audience will see blocky artifacting due to the reduced bitrate but there will be no dropped frames, freezing, or pausing. And if the ATEM is in a reduced bitrate mode due to this throttling, continue to monitor the upload and increase the bitrate if all is clear, then increase it some more, all the way back to the user-set bitrate.
PROBLEM: WHAT IF YOU'RE ALREADY AT THE LOWEST BITRATE?Well, you're not. The bitrate is defined in the Streaming.xml file, stored on the Mac at
- Code: Select all
/Library/Application Support/Blackmagic Design/Switchers/Streaming.xml
You can edit this file using a text editor (but not a word processor!) such as BBEdit or Atom. After making a backup of the original file, open the Streaming.xml file, and make a copy of the entire section that looks like this:
- Code: Select all
<profile>
<name>4.5Mb Low Bandwidth</name>
<config resolution="1080p" fps="60">
<bitrate>4500000</bitrate>
<audio-bitrate>160000</audio-bitrate>
<keyframe-interval>2</keyframe-interval>
</config>
<config resolution="1080p" fps="30">
<bitrate>3000000</bitrate>
<audio-bitrate>160000</audio-bitrate>
<keyframe-interval>2</keyframe-interval>
</config>
</profile>
Paste a copy of that section directly below the original section and give it a different "name", then define a lower bitrate, like this:
- Code: Select all
<profile>
<name>3Mb Low Bandwidth</name>
<config resolution="1080p" fps="60">
<bitrate>3000000</bitrate>
<audio-bitrate>160000</audio-bitrate>
<keyframe-interval>2</keyframe-interval>
</config>
<config resolution="1080p" fps="30">
<bitrate>2000000</bitrate>
<audio-bitrate>160000</audio-bitrate>
<keyframe-interval>2</keyframe-interval>
</config>
</profile>
Create more, even lower bitrate options so that you have the ability to go to very low bitrates if you need to while in a streaming session.
Save the file, relaunch the ATEM software, and you'll now see the newly defined "even lower" streaming settings in the pop-up menu where you select the streaming quality/bitrate. There are countless videos on how to modify the Streaming.xml file that go into more depth, but the above is all you need to know to create some lower bandwidth options for yourself.
HOW LOW CAN YOU GO?I tested various bitrates to analyze the effect on quality. I streamed to Mixcloud, with source video at 1080p 60fps. At 9Mbits/sec, the quality was of course excellent. I could see a bit of difference at 7Mbits/sec but pretty much the same high quality. At 4.5Mbits/sec any time there is a quick cross-fade instead of a cut between scenes, you can see artifacting. 4.5Mbits is usually the lowest setting. But going down to a custom-set 3Mbits/sec didn't look much worse. Going down to 2Mbits/sec was noticeably worse, though, as now panning against grass or a fine-patterned wall or carpeting showed significant smearing and muddy artifacting. One user on YouTube suggested going down to 1Mbit/sec was the only way he could stream on his low bandwidth connection. This works of course, and as long as you minimize pans, cross fades, and movement, 1Mbit/sec will actually work fine for talking head scenarios or any other low-movement video. It's better to have the option to drop down to 1Mbit and have smooth lower-quality video than to have a higher quality image that stutters and freezes constantly.
ATEM MINI PRO WISHLIST1) OBS is known for the ability to stream and record at independent bitrates. ATEM Mini Pro would be a better device if it could stream and record at independent bitrates. This would allow on-the-fly bitrate reductions in the livestream using the method described above, while keeping the recorded version at tack-sharp quality/bitrate.
2) Implementation of the Dynamic Bitrate described above, as an option vs relying on the cache, which is only good for momentary network congestion but doesn't solve longer periods of network congestion or unexpectedly impaired upstream bandwidth. For example, if another user on the same network decides to use Facebook live while you're streaming, your bandwidth available to the ATEM has effectively been halved. No cache is going to be able to deal with that, but a Dynamic Bitrate would handle it without issue.
3) So many streaming platforms expect 720p at 30fps -- ATEM Mini Pro should support that resolution, using its powerful hardware to transcode the 1080p video down to 720p. Since 1 frame of 720p video is 44% of the size of one frame of 1080p video, all of us streamers out there would have a much better experience since now the 4.5Mbits/sec of 720p video would be at a much higher -inherent- quality than 4.5Mbits/sec of 1080p video (not talking resolution here since 1080p is obviously higher resolution - talking about inherent quality due to artifacting).
I took the time to write this up in the hopes that it will help someone else.
Henry S. Kim
BIG FUN Disc Jockeys