Atem mini pro cache full

Questions about ATEM Switchers, Camera Converter and everything live!
  • Author
  • Message
Offline

GenePensiero

  • Posts: 27
  • Joined: Mon Aug 22, 2016 7:34 pm

Re: Atem mini pro cache full

PostThu Nov 04, 2021 6:36 pm

This may help, at least it has for my church and a friend's who was having the same issue.

In the streaming.xml file, when you tinker with the streaming profiles, it gives you two places per profile for bitrate.

Previously, for "high" I had set both of those parameters as the same value (450000). Then we would regularly have the cache problem.

My friend theorized that maybe those two bitrate values need to represent a range, rather than a single number.

I've changed the profiles for high, medium, and low to now be a variable range (pictured) and we haven't had the cache issue since. Neither has my other friend.

Screen Shot 2021-11-04 at 11.31.51 AM.png
Screen Shot 2021-11-04 at 11.31.51 AM.png (143.75 KiB) Viewed 6045 times
Offline

mploski

  • Posts: 2
  • Joined: Tue Nov 02, 2021 10:02 am
  • Real Name: Mateusz Płoski

Re: Atem mini pro cache full

PostFri Nov 05, 2021 3:23 pm

In my situation it only happens when I use 4g internet connection (reception is very good)
When I use cable internet it's perfect, there is no cache problems.
When I get 28 mb/s on the speedtest then I want and I should be able to stream stable at 6 mb/s. What is the problem? I realy don't understand this issue.
Offline

Dave Del Vecchio

  • Posts: 1603
  • Joined: Mon Nov 25, 2013 10:25 am

Re: Atem mini pro cache full

PostSat Nov 06, 2021 10:00 am

GenePensiero wrote:In the streaming.xml file, when you tinker with the streaming profiles, it gives you two places per profile for bitrate.

Previously, for "high" I had set both of those parameters as the same value (450000). Then we would regularly have the cache problem. My friend theorized that maybe those two bitrate values need to represent a range, rather than a single number.

I've changed the profiles for high, medium, and low to now be a variable range (pictured) and we haven't had the cache issue since.
The reason that there are two different bitrate configurations for each streaming profile is to better accommodate different video frame rates, since higher frame rates typically require a higher video data rate. In the default version of the XML file, the lower bitrate target for each streaming profile is used for frame rates up to 30 fps, and the higher bitrate target is used for frame rates up to 60 fps. The ATEM can easily pick which bitrate setting to use based on the video standard of the switcher and since each <config> element for a given <profile> in the XML file specifies a different fps value.

I am not exactly sure what happens if you specify two different <config> sections with the same fps value in a particular streaming <profile>. Presumably, the ATEM streaming engine will pick one of the two configurations to use, but I'm not sure how it decides which one.

If it turns out that the second <config> defined for the <profile> is the one that gets used in this situation, then this may be the reason why this improves the cache issue. You are essentially just streaming at a lower target bitrate, so it kind of makes sense that you are less likely to encounter bandwidth-related cache issues.
Offline

DutchRCTimeKeeping

  • Posts: 20
  • Joined: Mon Nov 01, 2021 8:23 am
  • Real Name: Mari van den Broek

Re: Atem mini pro cache full

PostSun Nov 07, 2021 12:46 pm

I have changed the bitrate as low as 1.5Mbps but as soon as I start streaming over 4G the cache starts to fill rapidly. Strange enough I have no problems streaming over 4G with OBS with 4500kbps or higher bitrate.
Offline

hbosch

  • Posts: 3
  • Joined: Sun Nov 07, 2021 6:27 pm
  • Real Name: Henry Bosch

Re: Atem mini pro cache full

PostSun Nov 07, 2021 6:59 pm

I also have LTE connection ,and also a LTE based private connection. 50/50 speeds, decent pings (under 70ms) to google.

We were doing fine for so long (like a year) and now things are unusable. We now use OBS studio from the ATEM Mini pro to get around the issue. Not sure what it is about the ATEM, but it can't seem to handle anything less than perfect latency?

I have been testing with other network gear as others have noted some issues with unifi. For those are LTE, are you also using ubiquity network gear?

**Edit** - Reproduced with other router/firewall as well.
Last edited by hbosch on Wed Nov 10, 2021 3:00 am, edited 1 time in total.
Offline

AtemTech971

  • Posts: 5
  • Joined: Sun Nov 07, 2021 8:53 pm
  • Real Name: Julien AMODEO

Re: Atem mini pro cache full

PostSun Nov 07, 2021 8:55 pm

Same probleme here, with an atem mini pro and an atem extreme iso.... Stream works fine since a long time but now... it's only work in WireCast and can't stream directly with the Atem.....
Offline

AtemTech971

  • Posts: 5
  • Joined: Sun Nov 07, 2021 8:53 pm
  • Real Name: Julien AMODEO

Re: Atem mini pro cache full

PostTue Nov 09, 2021 3:49 pm

I have found the probleme, it's not the atem it's the provider who block / limit the signal :s
Offline

GenePensiero

  • Posts: 27
  • Joined: Mon Aug 22, 2016 7:34 pm

Re: Atem mini pro cache full

PostWed Nov 10, 2021 7:07 pm

Dave Del Vecchio wrote:
GenePensiero wrote:In the streaming.xml file, when you tinker with the streaming profiles, it gives you two places per profile for bitrate.

Previously, for "high" I had set both of those parameters as the same value (450000). Then we would regularly have the cache problem. My friend theorized that maybe those two bitrate values need to represent a range, rather than a single number.

I've changed the profiles for high, medium, and low to now be a variable range (pictured) and we haven't had the cache issue since.
The reason that there are two different bitrate configurations for each streaming profile is to better accommodate different video frame rates, since higher frame rates typically require a higher video data rate. In the default version of the XML file, the lower bitrate target for each streaming profile is used for frame rates up to 30 fps, and the higher bitrate target is used for frame rates up to 60 fps. The ATEM can easily pick which bitrate setting to use based on the video standard of the switcher and since each <config> element for a given <profile> in the XML file specifies a different fps value.

I am not exactly sure what happens if you specify two different <config> sections with the same fps value in a particular streaming <profile>. Presumably, the ATEM streaming engine will pick one of the two configurations to use, but I'm not sure how it decides which one.

If it turns out that the second <config> defined for the <profile> is the one that gets used in this situation, then this may be the reason why this improves the cache issue. You are essentially just streaming at a lower target bitrate, so it kind of makes sense that you are less likely to encounter bandwidth-related cache issues.


Thanks for that clarification, Dave!

All I know is that we were getting the cache filling problem even when we were streaming at 1.5Mbps. I changed our streaming.xml file as noted above and now can stream at 4.5Mbps with no problem at all. Nothing else in our configuration or internet service has changed except that.
Last edited by GenePensiero on Thu Nov 11, 2021 9:30 pm, edited 1 time in total.
Offline

DutchRCTimeKeeping

  • Posts: 20
  • Joined: Mon Nov 01, 2021 8:23 am
  • Real Name: Mari van den Broek

Re: Atem mini pro cache full

PostThu Nov 11, 2021 6:53 pm

AtemTech971 wrote:I have found the probleme, it's not the atem it's the provider who block / limit the signal :s


Sure... why can I stream via OBS but not via ATEM both with the same provider?
Offline

AtemTech971

  • Posts: 5
  • Joined: Sun Nov 07, 2021 8:53 pm
  • Real Name: Julien AMODEO

Re: Atem mini pro cache full

PostThu Nov 11, 2021 7:40 pm

DutchRCTimeKeeping wrote:
AtemTech971 wrote:I have found the probleme, it's not the atem it's the provider who block / limit the signal :s


Sure... why can I stream via OBS but not via ATEM both with the same provider?


Same here, i don't understand, i've tryed a lot of thing, my provider was OK for Two years, and not it's not working directly with the ouput of the atem. I really don't understand and i need an solution because i don't want to stream with OBS/Wirecast.

What's your provider ? i'm in France ( Guadeloupe ) and i use ORANGE TELECOM.

With Another Sims from another Provider and same systeme, it's working...
Offline

redrob

  • Posts: 46
  • Joined: Fri Jun 07, 2013 5:56 am
  • Location: Buffalo, NY
  • Real Name: Rob Rothkopf

Re: Atem mini pro cache full

PostThu Nov 11, 2021 10:34 pm

DutchRCTimeKeeping wrote:
AtemTech971 wrote:I have found the probleme, it's not the atem it's the provider who block / limit the signal :s


Sure... why can I stream via OBS but not via ATEM both with the same provider?


Totally reasonable question.

Since this topic comes up frequently, I'll offer my standard response:

There is something about the VBR encoding on the ATEM, and how it is non-gracefully handling exception conditions which sends it into a non-recoverable tizzy. You are absolutely right -- if OBS can stream over a connection when the ATEM cannot, at the same bitrate to the same provider, the problem is with the ATEM, OR the provider's handling of a VBR stream. There are indeed differences.

Gotta ask -- since you said you're streaming over 4G (or was that someone else?) -- how is OBS connecting to the 4G network? How is the ATEM connecting to the 4G network? the same?
Offline

matthew003

  • Posts: 3
  • Joined: Wed Nov 17, 2021 12:14 am
  • Real Name: Matthew Randall

Re: Atem mini pro cache full

PostWed Nov 17, 2021 12:22 am

hi all,

something has seriously gone wrong in this area.

I have run dozens of streams both real and simple tests from home over 4g/5g connections using a 4g router and the atem mini pro plugged into the lan ports on the router.
i've run them at full "STREAMING HIGH" quality rate with ZERO issue for months now.

A couple of weeks back i set up at a wedding and the whole thing fall over instantly. Within 30 seconds the cache went to full.

Had to hack a whole buch of stuff in to get that stream online.

Since that day I have not been able to use the atem itself as the stream encoder at all without the cache filling up. I have tried multiple 4g routers, multiple 4g providers, high, medium, low quality and they all suffer.

I have been through every post in this forum too and nothing in here worked or applied to me.

I have only 20mb/s upload speed on 4g but i can't stream to youtube without it caching even at the lowest quality level.

However i plug the atem in via usb-c into a laptop running OBS, that laptop is connected to the same 4g router using ethernet (just like the atem) and OBS streams at 4.5mb/s with no dropped frames at all.

What has changed? Somethings going on here. THis is a lack of bandwidth. I've got plenty of it. and i've run streams in the past with much less with zero issues.
i'm tearing my hair out here.
Offline

matthew003

  • Posts: 3
  • Joined: Wed Nov 17, 2021 12:14 am
  • Real Name: Matthew Randall

Re: Atem mini pro cache full

PostWed Nov 17, 2021 1:41 am

Just a note that while the below situation did occur, i tried it again later than evening with a reboot and now that laptop too streams flawlessly. The ATEM switcher itself though, no good still.

here's a weird observation.

I noted earlier that no matter what I do with the atem itself, i cannot get it to stream any longer without massive packet loss over 4g or lte.
But when plugged into my macbook pro, using OBS, as a webcam only with obs connected via ethernet to my 4g router, it is flawless. doesn't drop a single frame.

That leads you to think that there is an issue with the atem itself.
But another test i ran was very interesting.

You see, my macbook pro is slightly under spec'd for this scenario. and i mean slightly. For the most part it sits at 20% cpu and zero% packet loss will streaming at 1080p50frames at 4000kbps.
Spins the fans up and drains the battery when not plugged in pretty quickly :-)

But i also have an HP elitebook windows 10 laptop with a slightly better CPU.
I connect the atem to that using USB. Connected that laptop to the 4g router over ethernet. Ran OBS. Started the stream at 4000kbps, 1080p50p and the OBS stream from this laptop hovers at around 40% lost frames.

This does lead you to think that there is something very odd in the way the ATEM and OBS on windows or maybe just windows itself or even just how that laptop handles packets over 4g.

I will continue to test this. Right now i'm running a stream over 4g using the router and the macbook and the atem as a webcam. flawless. Over 10 minutes now without a single dropped frame.
Let's try the windows laptop again...just to make sure that wasn't a misnomer.
Offline

DutchRCTimeKeeping

  • Posts: 20
  • Joined: Mon Nov 01, 2021 8:23 am
  • Real Name: Mari van den Broek

Re: Atem mini pro cache full

PostThu Nov 18, 2021 7:10 pm

redrob wrote:Gotta ask -- since you said you're streaming over 4G (or was that someone else?) -- how is OBS connecting to the 4G network? How is the ATEM connecting to the 4G network? the same?


Did test a few options (on different bitrates):

- ATEM Mini Pro connected directly to my phone by USB-C -- FAILED
- ATEM Mini Pro connected via LAN to my laptop who shares its internet via my phone as hotspot - FAILED

- ATEM Mini Pro connected via USB-C to my Laptop running OBS and my phone as hotspot -- WORKS LIKE A CHARM (low or high bitrates, no problems at all)...


So please BlackMagic, explain what is the difference between the RTMP-stream directly from ATEM Mini Pro to YouTube and the RTMP-stream from OBS to YouTube? Or should I return my ATEM Mini Pro and buy a ATEM Mini instead?
Offline

Daryl Toops

  • Posts: 52
  • Joined: Fri Aug 27, 2021 8:54 pm
  • Location: OR There
  • Real Name: Daryl Toops

Re: Atem mini pro cache full

PostThu Nov 18, 2021 9:52 pm

250 Mb/s upload speed

ATEM Mini Extreme ISO

Cat6 network and cables

Zero connection to Facebook. Nothing even shows up in preview to start the stream.

It worked without any problems back in Sept. October began the fail.

Tested all 6 output options, plus some custom ones.

Hacked the .xml to force Facebook to use alternative servers. It loads the .xml, but the settings blank out on ATEM and cannot stream.

Downgraded to 8.6.1, 8.6.3, no improvement.

Tested on YouTube, immediately connected and began streaming.

I suspect that Facebook changed a parameter and broke something. They did this 3-4 times in 2020 and didn't tell anyone. We lost streaming back then which required app updates to repair, so this may be their fault and not necessarily Blackmagic.
Resolve Studio 17.4.5, Threadripper 24-core, 64 GB RAM, 2 TB NVMe, Nvidia FE RTX 3080 10 GB. ATEM ME ISO.
Offline

AdmiralTriggerHappy

  • Posts: 5
  • Joined: Tue Nov 16, 2021 9:06 pm
  • Real Name: Andrew Wallace

Re: Atem mini pro cache full

PostFri Nov 19, 2021 12:24 am

I have the same problem with the cache

I'm running the Medium settings to twitch so its about 6Mbs and doing it over a 5G connection with the ATEM connected directly to the modem/router.

It works fine for awhile then something happens and the cache starts filling and the stream goes to crap and never seems to recover.

It works fine if I use OBS on my Macbook Pro, but for some reason I get an audio drift issue there which is as you can imagine makes it unworkable.

My 5G upload normally exceeds 10Mbs no problem.

As I post this I've got a test stream running and its currently fine, but I imagine its going to stuff up at some point.
Offline

AtemTech971

  • Posts: 5
  • Joined: Sun Nov 07, 2021 8:53 pm
  • Real Name: Julien AMODEO

Re: Atem mini pro cache full

PostSun Nov 21, 2021 5:34 pm

We all have the same problem strangely! What does BlackMagic do?
Nobody got an answer?
No update coming out, nothing from BlackMagic I have an Atem Mini Pro ISO and an unusable Atem Extreme ISO! Inadmissible!
If I just wanted a switcher I would have taken two Atem mini!
Help us BLACKMAGIC! Answer us! QUICK !
Offline

Raphaël Jacquot

  • Posts: 742
  • Joined: Wed Aug 29, 2012 4:15 pm

Re: Atem mini pro cache full

PostMon Nov 22, 2021 11:57 am

DutchRCTimeKeeping wrote:Had the same problem this weekend... cache filled-up rapidly... result: stuttering stream to youtube...

Tried again after lowering the bitrate in the streaming.xml file... same result and in most cases YouTube started complaining that I wasn't sending enough data...

Tried the same internetconnection with OBS and was able to normaly stream to YouTube in 1080p60...

I noticed that there is also a 720p option in the streaming.xml file, but no option to set my Mini Pro to stream at that resolution!

Should the build in encoder not be capable to stream lower bitrates or even op 720p?


I have the same problem when the MTU of the internet connection is smaller than 1500.
I suspect the (misguided / broken) IP stack in the atem to not work properly in that case.
Offline

NelsonYu

  • Posts: 1
  • Joined: Thu Dec 09, 2021 4:16 pm
  • Real Name: Nelson Yu

Re: Atem mini pro cache full

PostThu Dec 09, 2021 4:27 pm

I have the same issue started 3 months ago.

Not sure if BMD is reading this forum as their support rep believes that I am the only one experiencing this issue.
He basically requested me to lower my streaming quality, but it hasn't helped.
The internet connection is at over 700Mbps and latency is at 1ms.

We are now streaming OBS and works perfectly.
Really not sure what else to change.
Offline

Argi17

  • Posts: 3
  • Joined: Fri Apr 23, 2021 12:22 pm
  • Real Name: Oleg Serebriakov

Re: Atem mini pro cache full

PostFri Dec 17, 2021 8:40 am

I have the same issue.

But I think I found a problem. This is an Internet service provider.

The same device with the same stream settings and keys to YouTube works differently on different providers.

On one provider, the cache overflows 60 seconds after the start of the stream, on the other - it has been working for 5 hours.

I'm trying to figure out what network settings should be.
Offline

hbosch

  • Posts: 3
  • Joined: Sun Nov 07, 2021 6:27 pm
  • Real Name: Henry Bosch

Re: Atem mini pro cache full

PostSun Dec 26, 2021 4:40 pm

Hi All

Any of you using Ubiquity as your router? UDM, UDMPro?

See these threads for an identified issue. Customers with ATEM or Web Presenters.

https://community.ui.com/questions/Web- ... b8e?page=3

https://community.ui.com/questions/UDM- ... 7698dfa926

https://community.ui.com/questions/UDM- ... 9e3fdb0375
Offline

DutchRCTimeKeeping

  • Posts: 20
  • Joined: Mon Nov 01, 2021 8:23 am
  • Real Name: Mari van den Broek

Re: Atem mini pro cache full

PostFri Dec 31, 2021 8:29 pm

... I use no router at all (phone connected directly to Atem Mini Pro) -- cach full...
Offline

hbosch

  • Posts: 3
  • Joined: Sun Nov 07, 2021 6:27 pm
  • Real Name: Henry Bosch

Re: Atem mini pro cache full

PostSun Jan 02, 2022 4:24 pm

DutchRCTimeKeeping wrote:... I use no router at all (phone connected directly to Atem Mini Pro) -- cach full...


That's fair. In some conversation with others having the problem, it does also appear to be something the ATEM is doing vs other encoders. Silence from Blackmagic on the issue though. :cry:
Offline

AtemTech971

  • Posts: 5
  • Joined: Sun Nov 07, 2021 8:53 pm
  • Real Name: Julien AMODEO

Re: Atem mini pro cache full

PostSun Jan 09, 2022 10:23 pm

Not the same Year, but the same problem here.....
Offline

Daryl Toops

  • Posts: 52
  • Joined: Fri Aug 27, 2021 8:54 pm
  • Location: OR There
  • Real Name: Daryl Toops

Re: Atem mini pro cache full

PostMon Jan 10, 2022 7:03 pm

The solution for Facebook is to export the .XML file, open it up in an editor, and change the "username" and "password" to your account name, then save, and "restore" that to the ATEM. I suspect this may also work for other providers, and may be worth a try if nothing else is working.

Before I did this, there was zero preview of the stream on Facebook, the "On Air" button flashed which means a poor connection, and the cache was full within 1 minute. Afterwards: no problems, connected immediately, showed up in Facebook preview, was able to go live and others could view w/o issues, steady On Air light, and the cache stayed empty.

Update Jan 21. I did a 40 minute stream, with a 10-minute pre-show for recording. The show went live as a scheduled "event" without me having to do anything other than press the "ON AIR" button on the ATEM. The cache never went above 0. Successful connection to Facebook, no issues at all.
Last edited by Daryl Toops on Mon Jan 24, 2022 7:52 pm, edited 3 times in total.
Resolve Studio 17.4.5, Threadripper 24-core, 64 GB RAM, 2 TB NVMe, Nvidia FE RTX 3080 10 GB. ATEM ME ISO.
Offline

Raphaël Jacquot

  • Posts: 742
  • Joined: Wed Aug 29, 2012 4:15 pm

Re: Atem mini pro cache full

PostTue Jan 11, 2022 5:53 am

toddgreenlee wrote:Problem Solved!...at least for me.
I have had all the problems mentioned in this thread. Our ATEM worked fine for months then all of the sudden full cache problems, choppy video stream, loss of data, etc. I noticed that the ATEM was getting very warm and decided to put it on a laptop cooling pad. And, voila!, no more cache problems.


time to open it and remove all the dust bunnies
Offline

DutchRCTimeKeeping

  • Posts: 20
  • Joined: Mon Nov 01, 2021 8:23 am
  • Real Name: Mari van den Broek

Re: Atem mini pro cache full

PostFri Jan 14, 2022 9:33 pm

Dust is gathering on top of Atem Mini Pro's... BM needs to wake up and take some action...
Offline

sergey.ratz

  • Posts: 1
  • Joined: Sat Jan 15, 2022 11:08 am
  • Real Name: Serhii Ratz

Re: Atem mini pro cache full

PostSat Jan 15, 2022 12:37 pm

Seems to be no support from BM...
I see a lot of threads without any resolution...
And also no log information/tool for ATEM Pro for deeper diagnosis...
Offline

eheiger

  • Posts: 6
  • Joined: Sat Oct 31, 2020 5:50 pm
  • Real Name: Ed Heiger

Re: Atem mini pro cache full

PostMon Jan 17, 2022 3:44 am

me too...unplug the network cable from the atem for 2-3 seconds when cache starts to fill then plug it back in and it recovers every time. Not acceptable but it's the only way to make the streaming engine useable. have ticket open with support, they sort of hint engineers are aware so hopefully they are tuly aware. I provide feedback to them via the ticket to help them narrow down the problem. I'm convinced it's a bug, my internet has no issues when this happens. Have fios 300/300 and barely using it. Constant ping to my provider ny.castr.io showed no issues or changes when cache fills. When you take the network cable out it seems it resets the ip stack because cache will empty very quickly and the stream rolls along flawlessy afterwards.

Very frustrating

Ed
Offline

MarcusAlzona

  • Posts: 13
  • Joined: Tue Jul 31, 2018 10:52 pm
  • Location: Bethesda, Maryland / Washington DC Metro Area
  • Real Name: Marcus Alzona

Re: Atem mini pro cache full

PostSun Feb 27, 2022 6:27 pm

We are rapidly approaching the two-year mark for this (IMHO) critical ATEM bug.

In a thread started in July 2020, Henry Kim/@bigfundj wrote up a detailed analysis of the problem in October 2020:
https://forum.blackmagicdesign.com/viewtopic.php?f=4&t=117185#p675388

I added information in October 2020 as well, on how the ATEM Mini Pro ISO had a cascade failure during a wedding:
https://forum.blackmagicdesign.com/viewtopic.php?f=4&t=117185#p678105

In the *year and a half* since then, I’ve avoided using the ATEM in any streaming situations where I did not have a dedicated direct Ethernet connection at the venue, greatly limiting its usefulness as an event streaming platform. My multiple ATEM mobile rigs have become almost worthless.

I was very excited about the direct ATEM-Extreme-ISO-to-4G/5G-phone-connection feature….but the next time I tried it in May of 2021, it blew up in my face yet again….after streaming successfully for 20-30 minutes at some point it detected temporary network congestion, immediately filled the cache, locking up and causing failure of the stream AND the ISO recordings being done at the time.

I pulled out my iPhone and gimbal and got a stream and recording of the last 1/4 of the event, but all recordings of the middle half were lost forever, due to the complete failure of the ATEM.

I haven’t heard any updates addressing this problem from Blackmagic.
Raphaël Jacquot wrote:I have the same problem when the MTU of the internet connection is smaller than 1500.
I suspect the (misguided / broken) IP stack in the atem to not work properly in that case.
This seems like a really good theory on a potential key factor of the underlying problem. Doing a cursory search, I saw some people quoting 4G/LTE connections often having maximum MTUs in the 1400 and 1300 range
eheiger wrote:me too...unplug the network cable from the atem for 2-3 seconds when cache starts to fill then plug it back in and it recovers every time. Not acceptable but it's the only way to make the streaming engine useable. have ticket open with support, they sort of hint engineers are aware so hopefully they are tuly aware. I provide feedback to them via the ticket to help them narrow down the problem. I'm convinced it's a bug, my internet has no issues when this happens. Have fios 300/300 and barely using it. Constant ping to my provider ny.castr.io showed no issues or changes when cache fills. When you take the network cable out it seems it resets the ip stack because cache will empty very quickly and the stream rolls along flawlessy afterwards.
I did another event in the fall of 2021 using the ATEM Extreme ISO, and now that I think back at one point I did start getting the cache full syndrome, even though it was wired (but to a slightly congested Venue network. Thankfully, unplugging and re-plugging the Ethernet cable did eventually get the streaming cache un-frozen before total lock (and ISO recording loss) started to occur. But it really shouldn’t be this way!

Two years is a very very long time for this known critically fatal problem with a key functionality(streaming and recording) to go without being addressed. It is concerning that the issue can be documented and debugged, by Blackmagic customers and ATEM owners wishing the device to be successful, on the company’s user forums….but with no meaningful feedback or mitigation plans from the company.

It really feels like it’s being ignored or de-prioritized for some reason, and it shouldn’t be that the only solution for action is to document these issues in reviews on external vendor sites such as Amazon, B&H, etc. I hope that Blackmagic will try to address this issue ASAP.
Offline

dannymercadop

  • Posts: 1
  • Joined: Sat Mar 19, 2022 1:13 am
  • Real Name: Danny Mercado

Re: Atem mini pro cache full

PostSat Mar 19, 2022 1:16 am

I can't be more agree.

Same problem and same level of frustration.
Offline

Argi17

  • Posts: 3
  • Joined: Fri Apr 23, 2021 12:22 pm
  • Real Name: Oleg Serebriakov

Re: Atem mini pro cache full

PostSun Apr 10, 2022 1:12 pm

The problem is solved. At least I didn't have it anymore.

The problem is in the DNS of the Internet provider. Just configure DNS from Google in the network settings (4.4.4.4; 8.8.8.8).

It should work without failures.
Offline

AdmiralTriggerHappy

  • Posts: 5
  • Joined: Tue Nov 16, 2021 9:06 pm
  • Real Name: Andrew Wallace

Re: Atem mini pro cache full

PostSun Apr 24, 2022 10:42 am

Raphaël Jacquot wrote:
I have the same problem when the MTU of the internet connection is smaller than 1500.
I suspect the (misguided / broken) IP stack in the atem to not work properly in that case.


I wonder if the MTU size is the culprit, because sadly all my networks are running PPPoE and thus have a lower MTU thanks to the extra level of encapsulation.
Offline

michaelschutz

  • Posts: 9
  • Joined: Tue Aug 17, 2021 7:43 pm
  • Real Name: Michael Schutz

Re: Atem mini pro cache full

PostWed May 18, 2022 12:34 am

Just coming back to this with an anecdotal update for anyone who may still be having an issue.

I was thinking the issue at my office was related to the ISP-provided equipment, and it seems I was proved right. I updated to the 8.7 ATEM software and still was having problems. Got in touch with tech support wanting to do firmware updates to the main switch and router, because I can't do it locally for some reason. They did it recently and I just tested a little bit, and it all seems to be working properly now. I suspected my office setup because it worked fine at home so I was sure it wasn't a problem with the Mini Pro itself.

Just in case it's a help to anyone, we're on Shaw Business in Canada. If anyone else is having this issue, get your firmware updated and see what happens. It seems to have fixed it for us.
Offline

Justin_G8YTZ

  • Posts: 2
  • Joined: Sat Jun 25, 2022 12:36 am
  • Real Name: Justin Cockett

Re: Atem mini pro cache full

PostThu Jul 07, 2022 2:22 pm

I am having a very similar problem with an ATEM Mini Pro.

It is streaming 24x7 from a remote site and has been very stable, except when you connect to it via the ATEM software control and whilst the media files are downloading the stream buffer fills up. Quite often the Software control app then restarts the connection and the media file download occurs again only for the process to be repeated.

If I start a "ping" on the LAN local to the ATEM between a PC and the router gateway (normally 1ms response) the pings begin to fail. It looks like the ATEM is causing a local broadcast storm. I am going to try rate limiting the switch port to see if that helps.

On other occasions the stream crashes, never to be recovered without a power cycle, the ATEM Software Control stream button does nothing and neither does the physical button. You can prompt this behavior by uploading a new MACRO file, it seems to crash a couple of hours after uploading the file.

I wonder if the reason for this problem not being resolved is because its a hardware issue and the fix is physical surgery?
Offline

merlinbk

  • Posts: 2
  • Joined: Tue Dec 24, 2019 12:19 am
  • Real Name: Camilo Arevalo

Re: Atem mini pro cache full

PostSat Aug 20, 2022 3:28 am

What fixed the issue for me on YouTube was changing the default server address in the streaming xml to



<server>

<name>Primary</name>

<url>rtmp://a.rtmp.youtube.com/live2</url>

</server>

<server>

<name>Secondary</name>

<url>rtmp://b.rtmp.youtube.com/live2?backup=1</url>

</server>
Offline

Gaius619

  • Posts: 1
  • Joined: Fri Sep 16, 2022 1:03 pm
  • Real Name: Josh Nesbitt

Re: Atem mini pro cache full

PostFri Sep 16, 2022 3:53 pm

It sounds like there might be a couple different problems being discussed here. For everyone who is attempting to use the ATEM in conjunction with OBS, DO NOT press the Go Live button on the ATEM. Starting the broadcast and going live is handled through OBS, not the ATEM. If you Go Live on the ATEM as well as through OBS, you're trying to send two streams. When being used with OBS, the ATEM doesn't need to be connected to the network, only to the computer via USB C.
Offline

AdmiralTriggerHappy

  • Posts: 5
  • Joined: Tue Nov 16, 2021 9:06 pm
  • Real Name: Andrew Wallace

Re: Atem mini pro cache full

PostMon Sep 26, 2022 6:11 am

This is still a major issue!

I did a test stream to YouTube today and the cache filled up within about 15 minutes and took another 3 minutes after pressing the off button for the stream to empty and end the stream.

I was streaming with YouTube High Preset and my 60/30 connection was testing at 18 up before and after the stream (there is a decent amount of other traffic)

Now I'm going to change the server details as suggested by merlinbk and see what happens

Update, changing to RMTP from RTMPS had some effect but not as much
The cached filled up in about 15 minutes once again but it appears to have cleared faster this time.

I'll run some more tests to see what happens.
Offline

eMilty

  • Posts: 112
  • Joined: Mon Oct 22, 2018 9:54 pm
  • Location: The Netherlands
  • Real Name: Marco Miltenburg

Re: Atem mini pro cache full

PostMon Sep 26, 2022 11:40 am

I'm sorry to see that lots of people are still facing problems with this. BMD has been absolutely absent in supporting their devices. This problem has been around since the first Mini Pro and it present in all models that have streaming capabilities. It's even present in the Webpresenter models as shown by Dough Johnson in this YouTube video:



Even when you have a 1 Gb/s upload speed you will run into caching issues. So it's not just a too slow upload speed.

Just move away from BMD for streaming. I'm sorry to say, but it is just proven to be unreliable. Use OBS or buy something from Kiloview or Teradek. These products are known to work reliably in the same environment as where BMD products are having problems.

So far BMD has done nothing to make streaming any better or more reliable. Even a totally obvious color profile issue in the Webpresenter 4K has so far been unfixed. This should have been fixed within a week with a minor software update but has been present now for over a year. It's this lack of support that has forced me to look around for alternatives.
Offline
User avatar

Xtreemtec

  • Posts: 5149
  • Joined: Wed Jan 02, 2013 11:48 am
  • Location: The Netherlands

Re: Atem mini pro cache full

PostMon Sep 26, 2022 12:37 pm

Completely agree with Marco. It does not seem to have ANY priority with BMD to fix the REC2020 color space issues, Explain why Caching is rapidly filling and what can be done to prevent. And also as said by multiple users is that the Encoding is not that good at all.. :roll:

Our Vmix running FMLE always performed better in Encoding then the BMD Webpresenters..
We recently bought a Teradek Prism Flex. And wow the difference in quality between the Flex and Webpresenter 4K is a difference of DAY and NIGHT...

I'm Sorry to say but the Webpresenter is a low budget, low quality encoder.. Had much higher hopes..

And about Caching... It isn't your line speed... It's the Latency in your network / internet that is in our opinion the reason for the Caching issues. And since the streamers from BMD are not able to handle that proper, Even more if caching is running full, It often wont clear anymore until a reboot.. :roll:

We done 2 years of streaming with the Webpresenters HD and 4K.. Yes they work and we often had only between 3 and 6% of caching.. But we always be very carefull to have a clean network / internet line from the client. And internet in our Country is pretty reliable..

I wish BMD would spend a bit more time to explain certain things or make some tutorials on howto get the best results with hardware like this. That would fix a lot of issues for users.. ;)
Daniel Wittenaar .:: Xtreemtec Media Productions ::. -= www.xtreemtec.nl =-
4K OBV Trailer, ATEM TVS HD, 4M/E Broadcast Studio 4K, Constelation 8K, Hyperdeck Studio 12G, Ursa Broadcast 4K, 4K fiber converters with Sony Control
Offline

Doug Johnson

  • Posts: 58
  • Joined: Thu Feb 11, 2016 12:05 am

Re: Atem mini pro cache full

PostThu Sep 29, 2022 6:38 pm

I can actually weigh in a little bit more here since the video on my YouTube channel.

I used WireShark to capture the packets when this problem occurs. And I've worked with a couple networking professionals and we think we've worked out what is going on. And, unfortunately, it can't be fixed without some changes in the ATEM Mini/Web Presenter products.

What we're seeing is the Web Presenter over-reacting to dropped packets. Even a handful of dropped packets will cause the TCP window size to drop dramatically, which in turn limits how much data that the Web Presenter is willing to send before receiving acknowledgement from the RTMP server that it is streaming to. With TCP networking, whenever a packet is lost in transit the receiving server will request that it be re-sent by the sending device. Whenever this happens, the sending device will throttle back on how much data it will send before waiting for an acknowledgement that it has been received.

Some packet loss is completely normal and is expected. From what we were seeing in the packet captures, the amount of packet loss we saw when the problem occurs is not abnormally high, and doesn't impact other streaming devices that I tested (Teradek VidiU, OBS). What does happen, though, is that the Web Presenter over-reacts and cuts down much too dramatically on how much data it will send, resulting in really low bitrates even though the connection and receiving server can handle much more data. So where other streaming devices continue along just fine, the Blackmagic streamers basically give up and start buffering and dropping frames.

In my opinion, the Web Presenter and ATEM Mini are fundamentally broken in this way. Tweaking the TCP stack to not reduce the TCP window size so quickly would fix the problem, but it appears that they don't know how or are unwilling to fix it, or as far as I can tell, even acknowledge that it even exists.

The problem is more widespread than it appears. I've had any number of people reach out to me and tell me that they're having the same problem I am. It's much worse with higher bandwidth streams (4K, for example) but I've heard from more people who have an issue with HD video.
Offline

Doug Johnson

  • Posts: 58
  • Joined: Thu Feb 11, 2016 12:05 am

Re: Atem mini pro cache full

PostThu Sep 29, 2022 6:42 pm

...I will also mention that as we were doing testing, it looks like most of the packet loss we were seeing occurred once the data made it into Google's own networking infrastructure. So there was quite literally nothing we could do on our end to fix it.
Offline

eMilty

  • Posts: 112
  • Joined: Mon Oct 22, 2018 9:54 pm
  • Location: The Netherlands
  • Real Name: Marco Miltenburg

Re: Atem mini pro cache full

PostThu Sep 29, 2022 8:12 pm

Excellent research Doug and it confirms the problems we are all seeing. It's not a bandwidth issue but latency is an important factor as Daniel Wittenaar also said. Latency and TCP window size go hand in hand.

What is the initial TCP window size at the start? And to what size does it resize to when packets are lost?

I'm afraid that BMD might not even be able to fix this. They probably bought (a license for) a TCP/IP library for their Xilinx FPGA platform and this behaviour is implemented in this library. They might be able to tune/increase the initial TCP window size (if they have enough RAM for it) but by how much it resizes might not even be configurable. This library might also be intended for fairly simple TCP/IP connection with fairly low data throughput and not for streaming usage with 10 or even 20+ Mb/s for hours continuously. Still hopeful though they can and will fix this. If I would be boss of their software development this would be a top priority for me.
Offline

Ian Morrish

  • Posts: 578
  • Joined: Sun Jan 18, 2015 9:24 pm
  • Location: New Zealand

Re: Atem mini pro cache full

PostFri Sep 30, 2022 5:35 am

I have also seen on the Mini Pro that packet out of order error is returned from RTMP server but ATEM keeps sending the next packet so buffer just runs to full. Haven't retested recently after the last couple of updates. I was using my own RTMP server on Linux with trickle to simulate low bandwidth.
Regards,
Ian Morrish
Video Integrated Scripting Environment
(Windows PowerShell with ATEM driver + more)
https://ianmorrish.wordpress.com
Offline
User avatar

Xtreemtec

  • Posts: 5149
  • Joined: Wed Jan 02, 2013 11:48 am
  • Location: The Netherlands

Re: Atem mini pro cache full

PostFri Sep 30, 2022 11:36 am

Doug Johnson wrote:I can actually weigh in a little bit more here since the video on my YouTube channel.

I used WireShark to capture the packets when this problem occurs. And I've worked with a couple networking professionals and we think we've worked out what is going on. And, unfortunately, it can't be fixed without some changes in the ATEM Mini/Web Presenter products.

What we're seeing is the Web Presenter over-reacting to dropped packets. Even a handful of dropped packets will cause the TCP window size to drop dramatically, which in turn limits how much data that the Web Presenter is willing to send before receiving acknowledgement from the RTMP server that it is streaming to. With TCP networking, whenever a packet is lost in transit the receiving server will request that it be re-sent by the sending device. Whenever this happens, the sending device will throttle back on how much data it will send before waiting for an acknowledgement that it has been received.

Some packet loss is completely normal and is expected. From what we were seeing in the packet captures, the amount of packet loss we saw when the problem occurs is not abnormally high, and doesn't impact other streaming devices that I tested (Teradek VidiU, OBS). What does happen, though, is that the Web Presenter over-reacts and cuts down much too dramatically on how much data it will send, resulting in really low bitrates even though the connection and receiving server can handle much more data. So where other streaming devices continue along just fine, the Blackmagic streamers basically give up and start buffering and dropping frames.

In my opinion, the Web Presenter and ATEM Mini are fundamentally broken in this way. Tweaking the TCP stack to not reduce the TCP window size so quickly would fix the problem, but it appears that they don't know how or are unwilling to fix it, or as far as I can tell, even acknowledge that it even exists.

The problem is more widespread than it appears. I've had any number of people reach out to me and tell me that they're having the same problem I am. It's much worse with higher bandwidth streams (4K, for example) but I've heard from more people who have an issue with HD video.



Excellent info Doug. Did you send this out to a product manager / ( not support because support is just a wall to prevent too much data towards product managers and engineers.. )

I am in touch with a few of them.. I could forward this info if not done yet.. Maybe they are not even aware of the problem on a engineering standpoint..
Daniel Wittenaar .:: Xtreemtec Media Productions ::. -= www.xtreemtec.nl =-
4K OBV Trailer, ATEM TVS HD, 4M/E Broadcast Studio 4K, Constelation 8K, Hyperdeck Studio 12G, Ursa Broadcast 4K, 4K fiber converters with Sony Control
Offline

nov1ce

  • Posts: 1
  • Joined: Sat Oct 01, 2022 8:28 am
  • Real Name: Nigel Smith

Re: Atem mini pro cache full

PostSat Oct 01, 2022 9:44 am

Hi friends:

I’ve been running flawlessly for over a year and suddenly I’m having the same problem.
I’ve checked cables and bit rates and stream quality and updated software etc etc and no fix.
Read the whole thread.
Very frustrating for all concerned.

One thing I note is that the on air button flashes immediately indicating the stream isn’t even leaving the Atem to YouTube. Is this true for others too?

I’d love a fix or we’ll never go live again. Just record and upload.

Any new input received with thanks,

Nov1ce.
Offline
User avatar

Xtreemtec

  • Posts: 5149
  • Joined: Wed Jan 02, 2013 11:48 am
  • Location: The Netherlands

Re: Atem mini pro cache full

PostTue Oct 04, 2022 11:16 am

nov1ce wrote:One thing I note is that the on air button flashes immediately indicating the stream isn’t even leaving the Atem to YouTube. Is this true for others too?

This sounds more like you have no internet connection on the Atem anymore.. Is it running DHCP or did you fix it to an IP address?

We often do DHCP. As then you are sure the lease is renewed from the router with all correct settings.
Daniel Wittenaar .:: Xtreemtec Media Productions ::. -= www.xtreemtec.nl =-
4K OBV Trailer, ATEM TVS HD, 4M/E Broadcast Studio 4K, Constelation 8K, Hyperdeck Studio 12G, Ursa Broadcast 4K, 4K fiber converters with Sony Control
Offline

ReefNZ

  • Posts: 3
  • Joined: Sun Dec 05, 2021 6:52 am
  • Real Name: Reef Reid

Re: Atem mini pro cache full

PostThu Oct 06, 2022 6:54 am

atem-issue.jpg
atem-issue.jpg (102.74 KiB) Viewed 935 times
I am having the same issue and I have been for some time now. I just happened upon this thread and having read through it all, I can relate. I operate two ATEM Mini Extreme ISOs in two separate venues. Both are having the same issue. Both venues are music venues and one is used more than the other and is where I am seeing the issue more often than the other.

Some environmental context about the venue where the issue occurs more:
- Live stream events occur weekly and are approximately 1-3 hours in duration
- Content is streamed to You Tube
- Content is streamed at 1080p, 60fps
- Three PTZ cameras and four Marshall CV506 cameras are used in total
- a PC is used as the 8th input for titles etc via H2R as well as Companion for the Streamdeck XL
- The internet connection is delivered to the venue via fiber optic high speed broadband
- The Internet is provided as a PPPoE connection and offers up to 1000Mbps down and 500Mbps up and it easily achieves around 90% of these maximums (as expected) consistently
- The site router runs RouterOS (Mikrotik) and during the times of the issues there is nothing in the logs or traffic graphs that would explain this issues
- I've also setup my PC that is fitted with very capable hardware (Core i9-12900k CPU and NVIDIA RTX3090 GPU) running OBS connected to the USB-C port of the ATEM Mini Extreme ISO and when the ATEM encoder has the cache issue, the PC running OBS does not!
- The bitrate used on via OBS is much higher than the ATEM and in CBR and never skips a beat the whole time!
- I have yet to packet sniff the setup, but, at this point I think it is reasonable to assume the ATEM is at fault here and other posts in this thread seem to support that.
- When testing with the PC and the ATEM, these two devices are the ONLY devices on the network and they are hardwired via Ethernet directly to the gigabit interfaces on the router. There is no WiFi being used and the router is ONLY doing routing and firewall, no intensive tasks at all and the CPU/Mem usage proves that.

Question: From what I can tell the ATEM Mini Extreme ISO has not had a firmware update since April 2021? Version 8.6.1 is that correct? I'm not experienced in live streaming gear, but 18 months seems like a long time? My expectation is that there should be at least quarterly updates? How are issues in firmware addressed??

Diagram of the connectivity attached.

I've logged an email for this as I couldn't see how to actually log a support ticket?? It is my first time and my first post in this forum. I'm not holding my breath though having read this thread and seen that it has been going on for some time.

I am now using a capable laptop in both venues to remove the risk, but ultimately this is core and critical functionality that should be working and part of the cost of the pro/iso units. I don't particularly like VBR over CBR, but right now, I'd settle for a stable stream using the inbuilt encoder.

I am quite likely in a position in NZ to return this product to the local retailer that it was bought from and get a 100% refund under the guise that the Vendor has their head in the ground and that the failing component is a core part of the unit, but the problem is finding something in the same feature/price range as it.
Offline

ReefNZ

  • Posts: 3
  • Joined: Sun Dec 05, 2021 6:52 am
  • Real Name: Reef Reid

Re: Atem mini pro cache full

PostMon Oct 10, 2022 7:58 pm

Update:
I the BMD distributor/support folks in my area of the world (NZ) have been in contact and we have gone through all the typical things. Some more information, I also get the same issue whether I am streaming to YT or Facebook, but the venues both require these to go to YT.

The YT hosts for us are in Sydney Australia and round trips are approx 36ms which is more than ample for streaming. In fact the round trip for Facebook is approximately 15ms even better. I even used my PC that runs OBS, ramped it up to 50Mbps, then disconnected the Ethernet and connected it to the router via WiFi and it streamed solidly for 3 hours.

The BMD distributor has now requested the unit back (which is completely fair) which has to go from NZ to Australia and now means I have to find 2-3 weeks in the venues calendar to do so (because right now the workaround alleviates/masks the issue). But even if they replace it, I am not confident the issue will go away? They to date have been unable to re-produce the fault.

I can make this issue happen at will now, so while I am able to do this, I might packet sniff the connection and then also try my phone connected via USB and using 4G data to see what happens. Unless somebody else has already done this?
Offline

User2021

  • Posts: 4
  • Joined: Sun Oct 16, 2022 8:37 pm
  • Real Name: Peter Jacobs

Re: Atem mini pro cache full

PostSun Oct 16, 2022 8:45 pm

Sadly I am now following this discussion. We have been running our ATEM Mini Pro ISO flawlessly for 26 months and only today started experiencing connection issues with YouTube during a stream and cache issues on the ATEM.
What changed is that I upgraded my software for the first time since installing the ATEM.

Can I easily go back to the old version of code in one step?

Peter

Sent from my Pixel 4a using Tapatalk
PreviousNext

Return to Live Production

Who is online

Users browsing this forum: No registered users and 36 guests