Atem mini pro cache full

Questions about ATEM Switchers, Camera Converter and everything live!
  • Author
  • Message
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Atem mini pro cache full

PostMon Jul 06, 2020 7:41 am

Goodday,

I am the happy user of an Atem mini pro device. Switching and recording is flawless but with livestreaming I am experiencing some issues.

My setup:
- 4 input devices into switch
- 1 preview out
- usb-c to computer for control software
- Wired network to a router to send data to YouTube.

Connectivity test done on seperate device, connected to the router, 60mbit download / 40 mbit upload
When starting the stream, all seems fine, but after half a minute the cache starts filling up.

Bitrate is set to 4,5 mbit. The internet speed should be sufficient, but still, the cache fills up until full.

Cache full means lost data, which is noticeable in the stream, since a lot of frames are missing.


Anybody experienced the same? Any solutions?
For now a solution would be to connect the switch to device and use it as webcam input for the stream, but I rather do it over network.


Thanks!
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Re: Atem mini pro cache full

PostTue Jul 07, 2020 2:53 pm

Attached a picture of the cache which is filling up. This is approx 30 sec after starting feed to YouTube.
Note that the data rate is the same as set quality. Cache dump problem?
Attachments
E8DEE2EA-1EE2-406E-9379-468FE2B6F14C.jpeg
E8DEE2EA-1EE2-406E-9379-468FE2B6F14C.jpeg (51.25 KiB) Viewed 5162 times
Offline

RSDancey

  • Posts: 12
  • Joined: Thu May 14, 2020 5:14 pm
  • Real Name: Ryan Dancey

Re: Atem mini pro cache full

PostTue Jul 07, 2020 3:32 pm

Can you disconnect the USB cable and control the Mini via ethernet? The system may be sending some stream data to the computer and some data up the wire directly.
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Re: Atem mini pro cache full

PostTue Jul 07, 2020 3:36 pm

Will try, but cannot see how this affects the cache
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Re: Atem mini pro cache full

PostTue Jul 07, 2020 3:52 pm

RSDancey wrote:Can you disconnect the USB cable and control the Mini via ethernet? The system may be sending some stream data to the computer and some data up the wire directly.


I am just thinking about this. The switch was not connected to a pc with internet access. So it couldn’t send data away.

Also, another try it was not connected to pc, it was just a standalone without control panel open.
Offline

tmrufe

  • Posts: 5
  • Joined: Wed Jun 10, 2020 7:44 pm
  • Real Name: Timothy Rufenacht

Re: Atem mini pro cache full

PostTue Jul 07, 2020 4:06 pm

We're currently troubleshooting the same issue. It took a few weeks to get the RTMP stream to handshake/stream to BoxCast (we tried multiple different resets of the Streaming.XML file, with BoxCast's help/confirmation that our code was correct from the start, as well as a few other integrator friends we know just to get some other minds on it). It finally started working last week, but regardless of streaming quality set in the ATEM, it still fills the cache (not to mention what's going out is extremely poor quality), even though we're able to stream out OBS at full quality with no issues. We've checked the network/port configurations, and even moved the ATEM to the OBS Comp Ethernet that streams fine and the ATEM seems to be having issues.

OBS is set at 5Mb/s, and we've mostly be trying to test the ATEM at the 'Streaming Med' profile for 4.5Mb/s.

Our entire system is set at 1080p30 for consistency. Every step of the way we've gotten confirmation that we do, in fact, know what we're doing, but so far that hasn't solved any of our problems...
Offline

Rob Munzing

  • Posts: 2
  • Joined: Sat Apr 25, 2020 1:54 pm
  • Real Name: Robert Munzing

Re: Atem mini pro cache full

PostWed Jul 08, 2020 4:00 pm

We expierance the same problem with BoxCast so please respond with any further information they may share and thanks for this confirmation.
Offline

oliveramr

  • Posts: 1
  • Joined: Wed Jul 08, 2020 7:57 pm
  • Real Name: Oliver Munoz

Re: Atem mini pro cache full

PostWed Jul 08, 2020 8:03 pm

I have the same problem, some solution.

i try in youtube and facebook.
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Re: Atem mini pro cache full

PostThu Jul 09, 2020 8:06 am

tmrufe wrote:It finally started working last week, but regardless of streaming quality set in the ATEM, it still fills the cache (not to mention what's going out is extremely poor quality), even though we're able to stream out OBS at full quality with no issues.



Can you explain what you did to get it working?
Could it be a convert cache? so a cache which converts the video to the set quality?

It is strange that the outgoing data is reaching 4,5 mbit while the video is still going out broken... even though it reaches the set quality...

Seems that I am not the only one and more people are experiencing this issue. I've send a support request to BDM aswell, will revert here when I have a response.
Offline

Sfloor

  • Posts: 7
  • Joined: Mon Jul 06, 2020 7:34 am
  • Real Name: Silvan Floor

Re: Atem mini pro cache full

PostFri Jul 10, 2020 11:56 am

Problem solved! Or at least, problem moved!

I found out that the switch is uploading with 4,5mbit when I set the quality to 4,5 mbit. So it cannot be a network problem. When I noticed the cache filling up I started thinking about what the switch is doing. And than I came with the idea that the switch needs to convert the video output to a much lower quality. So the switch is totally capable of handling the upload, but is not capable of converting a 1080p video to a quality this low.

I've tested this with all settings as before, with the only change that I set the upload quality tot 9mbit.

No cache problems, stream was running smooth.


So, if your cache fills up and you reach your set quality with your uploadspeed, take a higher upload quality and problems will be solved (until you reach the capacity of your bandwith)

I've send this story to BMD aswell, awaiting their response.
I've searched a lot to find an answer to this problem, couldn't find any documentation on it.

I hope that the above helps others aswell.
Offline

JoshWise

  • Posts: 1
  • Joined: Sat Jul 11, 2020 2:38 am
  • Real Name: Josh Wise

Re: Atem mini pro cache full

PostSat Jul 11, 2020 2:50 am

Yes, please let us know what Blackmagic says.

I am having a similar issue, except I cannot get online whatsoever. On-Air flashes, but data rate is capped at .16 mbps and the Cache fills up over the course of 30 seconds. The live stream never reaches its destination either on all 6 data rate settings.

Have troubleshot the .xml file, key, rtmp, all of that to confirm I am entering everything correctly. Currently on network that has 150mbps and I'm not locked to any bandwidth and MTU is set to 1500 (which on another BM thread solved another guy's issue).

ATEM control works via all methods:
-network (separate ethernet cords to computer and Mini Pro)
-via usb from computer with ATEM Mini hooked up direct to network via ethernet (although can't use USB because we record using it)
-Computer hooked to network via ethernet and ATEM hooked up to computer via ethernet

AND all of these yield the same result: .16mbps and cache fills.

Any other insights appreciated, hopeful since this thread is fresh. TY.
Offline

Denny Smith

  • Posts: 12886
  • Joined: Thu Aug 01, 2013 4:19 pm
  • Location: USA, Northern Calif.

Re: Atem mini pro cache full

PostSat Jul 11, 2020 6:48 pm

Good catch, thanks for passing this along.
Cheers
Denny Smith
SHA Productions
Offline

koolkalangdude

  • Posts: 1
  • Joined: Wed Aug 05, 2020 7:44 am
  • Real Name: Sherwin So

Re: Atem mini pro cache full

PostWed Aug 05, 2020 7:49 am

Any update on the Cache Full of atem mini pro? We tried the 9mbit upload setting but still similar issue, cache full. Any help will be much appreciated.
Offline

Rileyvasquez

  • Posts: 1
  • Joined: Sat Aug 08, 2020 1:41 pm
  • Real Name: Riley Vasquez

Re: Atem mini pro cache full

PostSat Aug 08, 2020 1:49 pm

Any more luck or thoughts on this?

I was using an ATEM Mini Pro to stream and last night had a stream crash right before the show started. We tested the stream multiple times and didn’t have any issues. But sure enough, 5 minutes before the show started we started the stream with just a graphic and the cache filled up in about 30 seconds. Super frustrating!

I ended up using Blackmagic as just the switcher and letting OBS handle the stream...and it worked perfectly. It even made me wonder why use Blackmagic for a stream ever again!?

If anyone has more insight on this, I’d greatly appreciate the info.
Offline

newcityjim

  • Posts: 1
  • Joined: Fri Aug 21, 2020 6:12 pm
  • Real Name: James D Ward

Re: Atem mini pro cache full

PostFri Aug 21, 2020 6:15 pm

Hey Everyone,

I have had the same issue but my upload has been set at "Streaming Medium" or 7.0 Mb/s.

Any further insight into this problem?

Thanks!
Offline

mmadeiros

  • Posts: 1
  • Joined: Sun Sep 06, 2020 4:18 pm
  • Real Name: Matthew Madeiros

Re: Atem mini pro cache full

PostSun Sep 06, 2020 4:41 pm

Hello all,

Been having this same issue for the last few weeks. When the cache fills up, the connection to YouTube Studio starts dropping, and the stream starts falling apart. I've read on other threads that some users report very high operating temperatures. This has been my observation whether we use the unit indoors with AC, or outdoors. We've had to unplug the unit a few times when it became unresponsive, presumably from overheating. For example could not go "off air" from the unit or control software...simply had to unplug. This is being used outdoors (under a tent, but still 85+ degree air temp.). The issue of the cache filling up seems to loosely correspond to the hotter weather we've had the last couple of weeks. If anyone has any other thoughts, it would be much appreciated! Seems I am not alone with this frustrating issue.

Will try to find a decent laptop cooling pad and see if that makes any difference next week, and will try to report back!

Thanks much,
Matthew
Offline

pconnord9

  • Posts: 1
  • Joined: Fri Sep 11, 2020 8:37 pm
  • Real Name: Paul Connor

Re: Atem mini pro cache full

PostFri Sep 11, 2020 8:43 pm

Very keen to see any solutions for this
I have been streaming daily for 10-12 weeks without any issues.
Over the last 3/4 days we are now seeing cache full problems.
Nothing else has changed which is very confusing.
Streamed direct from Atem Mini Pro via Ethernet port to Castr.io same as many weeks before.
( 20mpbs upload speed tested on Ookla )

would appreciate any help or advice
Offline
User avatar

swallow

  • Posts: 32
  • Joined: Tue May 26, 2020 2:08 am
  • Location: NYC
  • Real Name: Ethan Doh

Re: Atem mini pro cache full

PostSat Sep 12, 2020 1:18 am

Save all settings to xml file and check audio rate.
Some of you may have different problem(s) but it was the problem I was having and after changing it to 320Kbs (and now 256Kbs), it cache was no longer filling.
Ethan Doh
Offline

Cubacane

  • Posts: 1
  • Joined: Sun Sep 13, 2020 3:46 pm
  • Real Name: Victor Labrada

Re: Atem mini pro cache full

PostSun Sep 13, 2020 3:49 pm

I've been having the same problem, whether I stream to YouTube or CASTR. But it seems to change by day. On Thursday I was able to stream for two hours with no problem. This morning I barely made it five minutes. Has anyone else had this issue resolved by the audio fix above? I don't want to mess with the code unless I HAVE to (like I did for CASTR).
Offline

mtbills

  • Posts: 5
  • Joined: Fri Sep 11, 2020 3:26 am
  • Real Name: Matthew Bills

Re: Atem mini pro cache full

PostWed Sep 16, 2020 3:10 am

I checked our XML file, and the audio settings were not the problem. No progress on troubleshooting why our cache fills up so quickly.
Offline

Brennyboy

  • Posts: 3
  • Joined: Wed Sep 16, 2020 3:12 pm
  • Real Name: Brendan Kerins

Re: Atem mini pro cache full

PostWed Sep 16, 2020 3:30 pm

I`ve been having the same problem with the Cache filling up. I was streaming to youtube at medium Bitrate then because the cache was filling up I decided to drop to the low 4.5mbits, but still had the same issue. I checked the upload speed of the internet and it was a good 20 mbits so the problem wasn`t with the upload speed. I also had a problem with going off air as it seemed freeze.
I was beginning to think that the problem was with Youtube but now I think that Blackmagic needs to Address this problem Pronto.
Offline

mtbills

  • Posts: 5
  • Joined: Fri Sep 11, 2020 3:26 am
  • Real Name: Matthew Bills

Re: Atem mini pro cache full

PostWed Sep 16, 2020 8:50 pm

Brennyboy: Your queue is full, so when you press stop, that does stop recording. However, the interface stays "on air" as it slowly empties the queue by continuing the upload. If you wait for the queue to empty (annoying that you can't see the exact queue percentage when it is under 40%), the system will then close the stream.

Given the number of users reporting the unexplained "queue full" behavior while streaming to YouTube, I agree that BlackMagic should take a more active role in troubleshooting this issue.
Offline

Brennyboy

  • Posts: 3
  • Joined: Wed Sep 16, 2020 3:12 pm
  • Real Name: Brendan Kerins

Re: Atem mini pro cache full

PostSat Sep 19, 2020 2:35 pm

Thanks Matthew, I sent an email to Blackmagic,here is the reply:
Hi Brendan,

I'm sorry to hear you're experiencing this behaviour, I'll do my best to assist you with this.

Our ATEM Mini Switchers have a small amount of internal memory called cache. If the network connection becomes slow enough that it cannot support the data rate required for the video stream, then the cache will fill with video frames to compensate. If the cache memory reaches 100%, then it will no longer be able to compensate for the lack off bandwidth.

You can monitor the cache status on the MultiView Output. If it does reach 100% then you will start to see performance issues occurring with the stream, it may start dropping to black or the stream might drop out completely. The On Air indicator in the ATEM Control Software may also start flashing to indicate a poor connection.

I would recommend connecting your ATEM directly to your internet router via ethernet, this will usually improve the performance of the stream as the ATEM should have access to more bandwidth. If the ATEM is connected to the internet over a WiFi network, then this could limit the bandwidth and cause the stream to become unstable.

I would also recommend adjusting the Stream Quality setting in the ATEM Control Software, please try the Medium or Low option to see whether this improves the behaviour. I'm asking you to try this because it sounds like the behaviour you're experiencing is bandwidth related. The cache will only fill up if the Internet connection isn't fast enough to handle the stream.

Regarding your internet and network connection speed, it needs to provide at least 9Mb/s to the ATEM Mini Pro for High Quality, 1080p60 streaming, if the network bandwidth drops below this then the cache will start to fill up. So for High Quality you will need 6 to 9 Mb/s, for Medium Quality 4.5 to 7 Mb/s and for Low Quality 3 to 4.5 Mb/s.

For advanced users, you could investigate the QoS (Quality of Service) settings on your router to prioritise streaming, if you can then this could free up some extra bandwidth for video streaming. Always consult with your IT or Network technician to help with this. Any changes to the router should be performed by a qualified technician.
Offline

rbwilliams67

  • Posts: 1
  • Joined: Sun Sep 20, 2020 2:21 pm
  • Real Name: Richard Williams

Re: Atem mini pro cache full

PostSun Sep 20, 2020 2:25 pm

Same problem here, used the mini pro successfully for a few weeks and now the cache is filling quickly. I have 24+ mbps upload speed so I don't think that is the issue
Offline

Brennyboy

  • Posts: 3
  • Joined: Wed Sep 16, 2020 3:12 pm
  • Real Name: Brendan Kerins

Re: Atem mini pro cache full

PostMon Sep 21, 2020 3:32 pm

Richard, do you think the problem could be with YouTube ?
Offline
User avatar

bmpeter

  • Posts: 10
  • Joined: Mon Aug 13, 2018 4:58 pm
  • Location: Croydon
  • Real Name: Peter B-Maraczi

Re: Atem mini pro cache full

PostTue Sep 22, 2020 9:48 am

2 weeks ago my direct streams to Youtube were fine, but same settings and gears on last week was choppy because of the cache was full quickly.
There wasn't any software update between my 2 streams.
So to me it looks like a Youtube issue, somehow.
Peter
Offline

toddgreenlee

  • Posts: 2
  • Joined: Sun Sep 27, 2020 2:23 pm
  • Real Name: Todd Greenlee

Re: Atem mini pro cache full

PostSun Sep 27, 2020 7:13 pm

Problem Solved!...at least for me.
I have had all the problems mentioned in this thread. Our ATEM worked fine for months then all of the sudden full cache problems, choppy video stream, loss of data, etc. I noticed that the ATEM was getting very warm and decided to put it on a laptop cooling pad. And, voila!, no more cache problems.
Attachments
IMG_1876.jpg
IMG_1876.jpg (417.77 KiB) Viewed 2000 times
Offline
User avatar

Berndalen

  • Posts: 7
  • Joined: Mon Jun 22, 2020 11:07 am
  • Real Name: Petter Berndalen

Re: Atem mini pro cache full

PostMon Sep 28, 2020 12:23 am

Dear colleagues and dear employees of Blackmagic Design!

I have about 30 paid streaming jobs in front of me. From the beginning of July until the beginning of September, my four BMPCC4k and ATEM Mini Pro worked perfectly.

In early September, I switched to the ATEM Mini Pro ISO. It worked well in the beginning but on the 11th of September I started to have problems. It started with a well-paid streaming job, when the recording of all ISO files suddenly stopped.
STOP RECORD could not be pressed at all. Nothing happened wherever I pressed. ATEM Mini had hung up.
Also the recording in three of four BMPCC4k stopped. Fortunately, one of the cameras was still recording a BRAW file.

In the editing of the concert I broadcasted, I had to settle for only one camera, during the last two songs.

I tried in every way to provoke this problem in my studio in the days that followed. But then the recording worked again.

Tonight, the first concert in a series of 10 concerts I was hired to livestream, I had exactly the same problem as mentioned in this thread!

"ON AIR" flashes and it is completely impossible to get the stream on youtube as well as facebook. Cache fills up and get full.

And yes. I have the Ethernet cable in ATEM Mini, my network settings are correct. My Macbook Pro over Wifi on the same network as the ATEM Mini. The bandwidth is 20 Mbit/s upstream.

I do EXACTLY the same as the last 16 streamed concerts.

But now it does not work at all.

This is incredibly embarrassing as I am hired by a client for big money and am solely responsible on behalf of the concert producer and the artists who perform the concert.

All equipment from Blackmagic Design was certainly not free, it worked flawlessly at first, but now it does not work at all. During these corona times, I have invested everything I have in this equipment and managed to get many jobs throughout the autumn that lie ahead.

Although the equipment records but does not stream, it unfortunately makes it completely unusable, for what it is supposed to be able to do.

Would anyone be so kind as to tell you what's going on ?!
Offline

toddgreenlee

  • Posts: 2
  • Joined: Sun Sep 27, 2020 2:23 pm
  • Real Name: Todd Greenlee

Re: Atem mini pro cache full

PostSun Oct 04, 2020 2:59 pm

Problem Solved!...at least for me.
I have had all the problems mentioned in this thread. Our ATEM worked fine for months then all of the sudden full cache problems, choppy video stream, loss of data, etc. I noticed that the ATEM was getting very warm and decided to put it on a laptop cooling pad. And, voila!, no more cache problems.


Yeah, never mind.
That solution worked for exactly one week and now does not. Which means it had to be a coincidence. The ATEM is considerably cooler, but I'm still having CACHE issues. I think it's beyond time for Blackmagic to address this.
Offline

bigfundj

  • Posts: 1
  • Joined: Sat Oct 10, 2020 6:59 am
  • Real Name: Henry Kim

Re: Atem mini pro cache full

PostSat Oct 10, 2020 7:06 am

INTRODUCTION/STATEMENT OF PROBLEM
This is my report and analysis of the ATEM Mini Pro's "Cache Full" problem. While streaming, the streaming status area should display OK to indicate 0% cache in use. Users have experienced circumstances where the cache % will appear and start to increase, reach 100% (at which point the cache status will display "FULL"), leading to "failure" of the stream. Making matters more stressful, pressing the OFF button to attempt to end the stream (so as to restart it), results in no feedback that the OFF button has been pressed, and the stream appears to be locked in the On Air position but with the warning that 100% of the cache is full. The operator feels that cutting power to the ATEM is the only way to regain control of the ATEM (it isn't, but I understand the panic and belief that this is the only way to regain control).

I experienced this twice on 9/19/2020 while streaming, once just prior to the start of a wedding ceremony, then again in the middle of the ceremony. After the first occurrence, I tried to stop the stream and ended up pulling the power cord as it appeared that the ATEM had "locked up" (it hadn't - more on that later). I reduced the outgoing bitrate from 7Mbits/sec to 4.5Mbits/sec (bandwidth had been measured prior to the event at 11Mbits/sec and I was connected via ethernet cable directly into the router). Because the second occurrence happened in the middle of the ceremony, cutting power to the ATEM would have been disastrous, so I relied on the fact that I was simultaneously recording using the ATEM's built-in record-to-flash-drive feature. Although this "saved the day" (I uploaded the recording during the reception so by the time I left the event, the "playback" was live), I wasn't sure what experience the remote audience had during the second incident. I ended up switching to OBS for the reception just to be safe. Anecdotally but importantly, another vendor stated that at the same time I experienced the second incident that he appeared to lose control of his internet-controlled smart lighting of the event. He felt that the network was offline for some period of time before he was "back online" and able to control his internet-controlled lighting once again.

This "cache full" issue has been discussed on the manufacturer's forum for months (including the thread where I am posting this) and has been a problem with many streaming hosts (I am streaming to Mixcloud but the problem has been experienced with every streaming platform).

Proposed solutions by users have included all of the following, none of which solve the problem:
- disconnect the control cable (USB)
- increase the outgoing streaming bitrate
- put the ATEM on a laptop cooler (user thought it solved the problem, then retracted as a fluke)
- increase the outgoing audio bitrate


EXECUTIVE SUMMARY
In Sept 2019, OBS v24.0 introduced the advanced option to stream using a "dynamically variable upstream bitrate". This resolved many issues related to experiencing "dropped frames" due to unpredictable network congestion. Dynamic Bitrate was a game changer. Now, all stuttering/pausing due to dropped frames was a thing of the past. Instead, OBS will ratchet down the quality (intentionally) during the timeframe when the upstream bandwidth is hampered, then resume the intended bitrate when the network is back to expected bandwidth. The idea is that seeing extreme artifacting is better than experiencing dropped frames. I agree.

By contrast, the ATEM Mini Pro handles poor upstream bandwidth by use of an onboard memory cache. If upstream bandwidth becomes constrained and the intended data cannot be sent, the data is re-routed to a modest-sized memory cache within the ATEM hardware. This cache is a FIFO (First In First Out); as the network "clears up", the backlogged data is sent out. While the constrained bandwidth is occurring, the remote audience sees dropped frames and freezing (pausing) of the video. Note that audio is prioritized as much as possible, so often the remote audience will hear segments of uninterrupted audio with a very low frame rate (extremely low sometimes, like 1 frame onscreen for several seconds, then another frame, and so on). Generally, the remote audience will continue to see an extremely poorly performing stream (as opposed to complete cessation). To the operator, however, it will feel like the stream has completely stopped (it hasn't but the quality of the stream is so poor as to be not much better than completely stopped).

After 6 consecutive hours of extensive testing of a wide variety of variables and observation platforms, I have characterized the problem and have devised methodologies to mitigate it, and to minimize the effect of the problem on the remote audience. Skip down to "solution" if you don't care about the testing and analysis.


TESTING AND ANALYSIS
The first step was to recreate the Cache Full problem. This proved to be harder than it sounds! Even with the ATEM set to 9Mbits/sec, I was unable to experience the Cache Full problem. My test network supports 11Mbits/sec upstream bandwidth quite reliably, so the ATEM happily streamed at 9Mbits/sec. To create the problem, then, I set several other computers on the same network to upload 1GB dummy files to offsite servers, simultaneously while streaming. This ended up being a highly reliable way to create the problem. I could now experience the Cache Full problem "upon demand" and thus run experiments.

First, to understand the remote audience experience during these times, I used my cell phone on a completely separate network (LTE cellular data) to observe the stream. I saw a stream with poor performance - lots of frozen frames and pausing of the video. I was surprised to observe that for long stretches of time, the audio played smoothly even with an extremely low video frame rate. The ATEM appears to prioritize sending audio since it requires much less bandwidth. The stream was effectively unwatchable, however, due to the erratic frame rate and long moments of frozen frames.

What happens when the OFF button is pressed on the ATEM to stop the stream? As many have observed, it appears that nothing happens. There is no visual indicator that the OFF button has been pressed, and the operator appears to lack control of the ATEM hardware or the software controller. In other words, if the intent is to stop the stream, reduce the streaming bitrate, then start the stream, this appears to be impossible to execute because the ATEM (hardware and software control) appears completely unresponsive.

Not so fast. It turns out that this is an exercise in patience. When the OFF button is pressed to stop streaming, the ATEM faithfully stops attempting to fill the cache. In effect, anything happening "live" from the moment you press the OFF button CEASES to be captured by the ATEM. (Note that if using the Record function to record to a connected flash drive, the recording does continue, unaffected by all of this.) But the reason the OFF button doesn't appear to work ("ON AIR" continues to be lit no matter how many times you press OFF) is because the ATEM is faithfully continuing to pursue streaming of the content stored within the cache. If you wait long enough, you'll see the cache indicator drop from FULL to 99% then gradually work its way down all the way to 0%. Once the cache has emptied out to the stream, the OFF button lights up and the operator once again has control. You can change the streaming bitrate at this point, and you can press the ON AIR button again. Because the emptying of the cache can take a long time (in many cases it took 2 minutes during my testing) most operators will "give up" before that length of time, thinking the unit has become completely unresponsive and simply pull the power and reboot the unit. This it not advised as the remote audience gets a very abrupt halt of the stream.

So what happens if you wait out the emptying of the cache, reduce the bitrate, then go ON AIR again? I'll illustrate with a sample timeline:
1:00:00 cache starts to fill up
1:00:30 cache hits full within 30 seconds; operator presses OFF on ATEM (or in software)
1:00:31 no more video enters the cache, cache starts to empty out, but ATEM still shows "ON AIR"
.
.
.
1:01:30 a full minute later, the cache has emptied out, OFF button finally lights up
1:01:31 operator reduces bitrate, then presses ON AIR again
1:01:32 stream resumes (at the now lower bitrate)

From the remote audience viewpoint, this is what they saw:
1:00:00 stream starts to freeze and pause
...it takes a minute and a half to play the 30 seconds of live footage from 1:00:00 to 1:00:30...
1:01:32 stream resumes, but the footage from 1:00:30 through 1:01:31 was never seen

Clearly, 60 seconds of missing live footage is quite undesirable. The remote audience endures 90 seconds of a very poor quality stream that only covers 30 seconds of live footage, then when things resolve, we've "jumped forward" a minute, and the missing 60 seconds is never seen.


SOLUTION
The natural tendency of the operator when observing the cache filling up, is to hope that the % number will go back down and the problem will resolve itself. Don't wait. If the cache gets above 40%, press OFF immediately. I've found that it takes much less time to empty the cache if you do so at 40% than if you wait until it's FULL. You'll regain control of the ATEM within mere seconds as opposed to waiting a full minute to regain control. To clarify, as soon as that cache gets above 40%, press OFF, wait a few seconds and the OFF light will light up acknowledging the stream has been stopped and the cache has emptied, then lower your outgoing bitrate, then press ON AIR and now you're streaming again and your remote audience has only missed the few seconds of live footage while you were waiting for the ATEM to empty the small amount in the cache. I tested this repeatedly both ways, and emptying out a 40% cache is reasonably quick, as opposed to emptying a FULL cache. Remember, since the network is apparently unable to keep up with the bitrate, a FULL cache is not just two and a half times a 40% cache. It's much worse because it's having trouble streaming anyway, and now it has to endure that trouble for 2.5 times the amount of DATA which itself was uploading at a slower rate so will take much longer than 2.5 times the amount of TIME. Think of it like a clogged bathtub drain - if you detect the clog early, you can shut off the water supply and the small amount of water in the bathtub will drain reasonably fast through the clogged drain. But if you let the bathtub fill completely with water, it's going to be a very long wait for that huge volume of water to drain through the clogged drain. Similarly, it's easy to fill the cache, but much harder for the cache to empty through a clogged network connection.

Definitely have a flash drive plugged into the ATEM and be recording simultaneously so that you have a recording of the footage that ended up missing from the livestream.

Is this a perfect solution? Of course not. But it does minimize the problem to the remote audience, and it does minimize the amount of "missing footage" not seen by the remote audience.


RECOMMENDATION TO BLACKMAGIC DESIGN
Recognize that the solution given above is a "poor man's" version of what OBS is doing with its Dynamic Bitrate option. You stop the stream (as soon as the problem starts happening), reduce the bitrate, then go back on air. BlackMagic Design could fix this problem by implementing a Dynamic Bitrate option similar to the one in OBS. It's not difficult for BlackMagic Design to do so, because they're ALREADY detecting that an overflow condition is occurring because they're filling the onboard cache and reporting to the operator that they're doing so! So instead of filling up the cache, why not just reduce the bitrate within the software, monitor the buffer overflow condition, and if it's still backlogged reduce the bitrate further, and so on. The remote audience will see blocky artifacting due to the reduced bitrate but there will be no dropped frames, freezing, or pausing. And if the ATEM is in a reduced bitrate mode due to this throttling, continue to monitor the upload and increase the bitrate if all is clear, then increase it some more, all the way back to the user-set bitrate.


PROBLEM: WHAT IF YOU'RE ALREADY AT THE LOWEST BITRATE?
Well, you're not. The bitrate is defined in the Streaming.xml file, stored on the Mac at
Code: Select all
/Library/Application Support/Blackmagic Design/Switchers/Streaming.xml


You can edit this file using a text editor (but not a word processor!) such as BBEdit or Atom. After making a backup of the original file, open the Streaming.xml file, and make a copy of the entire section that looks like this:

Code: Select all
         <profile>
            <name>4.5Mb Low Bandwidth</name>
            <config resolution="1080p" fps="60">
               <bitrate>4500000</bitrate>
               <audio-bitrate>160000</audio-bitrate>
               <keyframe-interval>2</keyframe-interval>
            </config>
            <config resolution="1080p" fps="30">
               <bitrate>3000000</bitrate>
               <audio-bitrate>160000</audio-bitrate>
               <keyframe-interval>2</keyframe-interval>
            </config>
         </profile>

Paste a copy of that section directly below the original section and give it a different "name", then define a lower bitrate, like this:

Code: Select all
         <profile>
            <name>3Mb Low Bandwidth</name>
            <config resolution="1080p" fps="60">
               <bitrate>3000000</bitrate>
               <audio-bitrate>160000</audio-bitrate>
               <keyframe-interval>2</keyframe-interval>
            </config>
            <config resolution="1080p" fps="30">
               <bitrate>2000000</bitrate>
               <audio-bitrate>160000</audio-bitrate>
               <keyframe-interval>2</keyframe-interval>
            </config>
         </profile>

Create more, even lower bitrate options so that you have the ability to go to very low bitrates if you need to while in a streaming session.

Save the file, relaunch the ATEM software, and you'll now see the newly defined "even lower" streaming settings in the pop-up menu where you select the streaming quality/bitrate. There are countless videos on how to modify the Streaming.xml file that go into more depth, but the above is all you need to know to create some lower bandwidth options for yourself.


HOW LOW CAN YOU GO?
I tested various bitrates to analyze the effect on quality. I streamed to Mixcloud, with source video at 1080p 60fps. At 9Mbits/sec, the quality was of course excellent. I could see a bit of difference at 7Mbits/sec but pretty much the same high quality. At 4.5Mbits/sec any time there is a quick cross-fade instead of a cut between scenes, you can see artifacting. 4.5Mbits is usually the lowest setting. But going down to a custom-set 3Mbits/sec didn't look much worse. Going down to 2Mbits/sec was noticeably worse, though, as now panning against grass or a fine-patterned wall or carpeting showed significant smearing and muddy artifacting. One user on YouTube suggested going down to 1Mbit/sec was the only way he could stream on his low bandwidth connection. This works of course, and as long as you minimize pans, cross fades, and movement, 1Mbit/sec will actually work fine for talking head scenarios or any other low-movement video. It's better to have the option to drop down to 1Mbit and have smooth lower-quality video than to have a higher quality image that stutters and freezes constantly.


ATEM MINI PRO WISHLIST
1) OBS is known for the ability to stream and record at independent bitrates. ATEM Mini Pro would be a better device if it could stream and record at independent bitrates. This would allow on-the-fly bitrate reductions in the livestream using the method described above, while keeping the recorded version at tack-sharp quality/bitrate.

2) Implementation of the Dynamic Bitrate described above, as an option vs relying on the cache, which is only good for momentary network congestion but doesn't solve longer periods of network congestion or unexpectedly impaired upstream bandwidth. For example, if another user on the same network decides to use Facebook live while you're streaming, your bandwidth available to the ATEM has effectively been halved. No cache is going to be able to deal with that, but a Dynamic Bitrate would handle it without issue.

3) So many streaming platforms expect 720p at 30fps -- ATEM Mini Pro should support that resolution, using its powerful hardware to transcode the 1080p video down to 720p. Since 1 frame of 720p video is 44% of the size of one frame of 1080p video, all of us streamers out there would have a much better experience since now the 4.5Mbits/sec of 720p video would be at a much higher -inherent- quality than 4.5Mbits/sec of 1080p video (not talking resolution here since 1080p is obviously higher resolution - talking about inherent quality due to artifacting).


I took the time to write this up in the hopes that it will help someone else.

Henry S. Kim
BIG FUN Disc Jockeys
Offline

zendrum

  • Posts: 2
  • Joined: Mon Oct 05, 2020 10:35 pm
  • Real Name: Eric Sills

Re: Atem mini pro cache full

PostMon Oct 12, 2020 1:30 am

I am also having cache issues with both of my ATEM units. One is an ATEM Mini Pro, and the other is an ATEM Mini Pro ISO.

I ran a number of streams with no problems using both units. I then all of a sudden began having Cache full issues with many stream dropouts. The dropouts are in the video signal, but not the audio.

I have been watching this thread with interest to see if any solutions are found, and after reading toddgreenlee's post's, I also tried a cooling platform with fans. I attached a picture of my setup.

I ran a stream a couple days ago with all inputs running, and recording all ISO inputs at the same time. The ATEM ran flawlessly. No cache issues.

This evening I ran a stream with the exact same setup, but I forgot to plug in the fans to the USB adapter, and I immediately had cache full issues that spoiled the stream video. I interrupted the stream and switched to the non-ISO unit, but the cache filled up quickly on the other unit also.

This has me wondering if it really is a heat issue after all. I have a bunch more streams coming up, and I will absolutely remember to plug the fans in! I'll also have a laptop running OSB at the ready as a backup if necessary.

I'll report back soon and let this thread know what happens.

Does anyone at Blackmagic have any helpful words of wisdom yet?
Attachments
KIMG0011.jpg
KIMG0011.jpg (378.35 KiB) Viewed 1183 times
Offline

zendrum

  • Posts: 2
  • Joined: Mon Oct 05, 2020 10:35 pm
  • Real Name: Eric Sills

Re: Atem mini pro cache full

PostSat Oct 17, 2020 6:46 am

The cooling system didn't help me either.

Last week I had a stream fail immediately. Last night I had a stream run flawlessly for 2 hours.

Tonight I had a stream fill the cache right after hitting the button. We had to kill the stream snd then we went into panic mode connecting and disconnecting everything in the system. I spent a few minutes trying to do an emergency changeover to Facebook through the computer, but couldn't get the browser to recognize the ATEM as a camera. After much cursing, we gave up and reconnected the exact same setup we started with, hit the go live button and itt proceeded to run perfectly for an hour and a half.

What the sam hill is going on? I'm totally stumped.
Offline

RenewalTC

  • Posts: 1
  • Joined: Sun Oct 18, 2020 2:56 pm
  • Real Name: Todd E Cooper

Re: Atem mini pro cache full

PostSun Oct 18, 2020 3:05 pm

Our Church has two Atem Mini Pros and we have been having problem's with the cache filling up and breaking the stream. It's definitely related to temperature as it gets worse as the Atem heats up. We used a Flir thermal camera and the temperature on the outside of the unit is 130.7 degrees F. The cache is pegged at 99% and the stream is unwatchable.
Offline

MarcusAlzona

  • Posts: 8
  • Joined: Tue Jul 31, 2018 10:52 pm
  • Location: Bethesda, Maryland / Washington DC Metro Area
  • Real Name: Marcus Alzona

Re: Atem mini pro cache full

PostTue Oct 20, 2020 5:14 am

Berndalen wrote:Dear colleagues and dear employees of Blackmagic Design!

I have about 30 paid streaming jobs in front of me. From the beginning of July until the beginning of September, my four BMPCC4k and ATEM Mini Pro worked perfectly.

In early September, I switched to the ATEM Mini Pro ISO. It worked well in the beginning but on the 11th of September I started to have problems. It started with a well-paid streaming job, when the recording of all ISO files suddenly stopped.
STOP RECORD could not be pressed at all. Nothing happened wherever I pressed. ATEM Mini had hung up.
Also the recording in three of four BMPCC4k stopped. Fortunately, one of the cameras was still recording a BRAW file.

In the editing of the concert I broadcasted, I had to settle for only one camera, during the last two songs.

I tried in every way to provoke this problem in my studio in the days that followed. But then the recording worked again.

Tonight, the first concert in a series of 10 concerts I was hired to livestream, I had exactly the same problem as mentioned in this thread!


This really sounds like a nightmare scenario....after you doing all the prep work, testing, and setting up four BMPCC4k's that are multiple times the cost of the ATEM Mini Pro each....only to have it fail.

In the weeks/concerts since then, were you able to find a solution? Or did you simply give up on the ATEM Mini Pro's streaming functionality? I was contemplating adding a BMPCC4k to my ATEM Mini Pro/Pro ISO setups, but experienced this cache problem a week ago and now am wondering if this is the wrong path and wrong product line to place all my eggs....
Offline

MarcusAlzona

  • Posts: 8
  • Joined: Tue Jul 31, 2018 10:52 pm
  • Location: Bethesda, Maryland / Washington DC Metro Area
  • Real Name: Marcus Alzona

Re: Atem mini pro cache full

PostTue Oct 20, 2020 6:00 am

bigfundj wrote:INTRODUCTION/STATEMENT OF PROBLEM
This is my report and analysis of the ATEM Mini Pro's "Cache Full" problem. While streaming, the streaming status area should display OK to indicate 0% cache in use. Users have experienced circumstances where the cache % will appear and start to increase, reach 100% (at which point the cache status will display "FULL"), leading to "failure" of the stream. Making matters more stressful, pressing the OFF button to attempt to end the stream (so as to restart it), results in no feedback that the OFF button has been pressed, and the stream appears to be locked in the On Air position but with the warning that 100% of the cache is full. The operator feels that cutting power to the ATEM is the only way to regain control of the ATEM (it isn't, but I understand the panic and belief that this is the only way to regain control).

....(incredibly detailed write-up here)...

I took the time to write this up in the hopes that it will help someone else.

Henry S. Kim
BIG FUN Disc Jockeys


This was very helpful and insightful, both in stating a similar experience I had recently during a family wedding, and for the discussion of the probably causes and possible mitigations for the user and fixes for BlackMagicDesign.

When experiencing this issue, I responded in a few different manners in an attempt to fix the problem. Initially, it was 10 minutes before the start of the event, so when it started to fill, and the "stop stream" button seemed to be completely unresponsive, I first stopped the recordings then pulled the power to the ATEM Mini Pro ISO. When I brought it back up I reduced it to Stream Low (I believe)....and when it happened again after the start of the event, and I hit stop and nothing happened for a couple of minutes, I pulled the ethernet cable (was recording ISO to an SSD, so kept on the power).....that seemed to unblock the stream within seconds. Of course, it just started happening again.

Now, this outdoor location didn't have the best network, and I tried multiple network methods (WiFi Hardware Bridge, WiFi-Laptop-share, LTE-bridge). But I also experienced this at home, with the same methods...and same problem (eventually).

In the end, at the event, I ended up streaming the second half using my iPhone on a gimbal over LTE, using the ATEM Mini Pro ISO to capture the video off of the different cameras I was still running/transitioning, later trying to salvage the video from those edits (which worked out).

However, I'm not sure if I can trust the ATEM Mini Pro/ISO anymore for streaming, which negates a major value over a single camera iPhone streaming setup. Furthermore, my recent purchase of a Pro ISO was explicitly to capture the ISO tracks to an SSD for future editing.....but if the only way I can do a stable stream is to hook it to a laptop via USB-C, then there's no real point to having the ISO (and the ATEM Mini Pro's only differentiation is the multiview).

So instead of expanding my BMD equipment inventory with a Pocket Cinema 4k or two, I might be reducing it by returning my ATEM Mini Pro ISO at this point. Hopefully, they'll come out with a fix soon.

But one fear is that if the problem/fix is the Dynamic Bitrate suggestion you had......what if this is a hardware-level implementation for the ATEM...so not fixable via a firmware update....would need to wait until the "ATEM Mini Pro ISO Max"(my made up name) that has dynamic streaming and two HDMI outs...so multiview and program....or dual streaming destinations...so Facebook and YouTube....or one service + a local streaming bridge.


Marcus

P.S. To be fair...could any of these factors be the cause of these ATEM Mini Pro/ISO streaming problems? :

  • Adjustments to Facebook & YouTube's ingestion interfaces, making them less tolerant...or just different in a way that messes up the ATEM Mini Pro's streaming...
  • Increased streaming traffic/congestion due to worldwide distance learning for many local school districts and universities.
  • Networking optimizations by the large ISPs for streaming content...ISP streaming optimizers accidentally optimizing the ATEM's data, transparent nginx proxies, etc.

Anyway, hoping for the best.....let us know if you've had any progress or updates....
Offline

TemporalKinetics

  • Posts: 11
  • Joined: Thu Aug 13, 2020 1:51 pm
  • Real Name: Kevin Liesner

Re: Atem mini pro cache full

PostWed Oct 21, 2020 4:46 am

Thank you @bigfundj

I wrote to Blackmagic and asked about the full buffer (in our case when we called Spectrum, they did say there were intermittent issues in the area) and about not being able to stop it and asked if 8.5 fixes it.

Their reply:
"You are correct. If Internet connection is loosing or there is no enough bandwidth, the ATEM buffer will be overflow. Select a bit rate according to the capabilities of the Wi-Fi and internet upload speed. General rule of thumb is less than half your upload capability.For example if upload speed is 10 Mb/c, you can use Streaming Low 3 to 4.5 Mb/s in the ATEM Software Control.

The most recent version of Software includes all improvements and fixes. "

No info about not being able to stop it, no info about whether 8.5 fixes it and based on posts here, won't. Come on Blackmagic!

I streamed fine since July, so I'm inclined to believe that it is bandwidth related this time, though I now have OBS setup as a backup. I'm guessing I'll need to tell YouTube not to automatically stop the stream in order to be able to switch to it.

I will say you do need to be careful with heat. I placed this on top of a Hyperdeck Mini and it caused the Hyperdeck mini to fail recording.

Everybody is trying to get the latest junk out the door and no one (Microsoft, et al as well) is testing very well.
Offline

Ruebyi

  • Posts: 1
  • Joined: Fri Oct 30, 2020 11:10 am
  • Real Name: Tobias Lehmann

Re: Atem mini pro cache full

PostFri Oct 30, 2020 11:24 am

I ran into this issue during a livestream after 70 Minutes of streaming to Facebook

After Updating to 8.5.1 I was able to stream 90 minutes to Facebook in 9mbit/s

I will test further with 8.5.2 and other qualities...

Return to Live Production

Who is online

Users browsing this forum: No registered users and 41 guests