BRAW and RED RAW issue

Hello,

I have a nice Windows workstation with dual 3090' GPUs, Threadripper CPU, 256 gigs of RAM, and a RAID 0 with several M2 drives as my primary work drive. I do some VR work and you need all the power you can get to work with 8K 3D 360 footage. I can render all day long without issue using Mistika VR and this rig.

I have also edited lots of footage on this rig in Resolve from Sony A7's, stuff that doesn't shoot RAW. Content recorded in the A7 (MP4) works fine as well as content recorded to a Ninja V in ProRes.

Recently I had a project with RED Komodo footage shot and edited in RED RAW. While color grading in Resolve this footage caused the system to become unstable and crash. Like a HARD crash, whole computer shuts off crash, ha ha.

This is the only time this workstation has ever crashed like that. After some playing around I unplugged power to the second GPU (that wasn't attached to my display) and that fixed it. I lost processing speed, but at least it was stable.

While playing with another project using Pocket 6K's shooting in BRAW, the same thing happened. Using both GPUs would hard crash the system. Unplugging one of the GPUs would fix it.

Any ideas on how to fix this? Real bummer to lose a GPU and slow down. Seems to only be a RAW file issue with Resolve.

Is there a setting or preference to change? Something under memory and GPU or decode options?

Thanks,
Rich

Edit: Seems that the issue was my 1,300W PSU. Upgrading to a 1,600W PSU has fixed the issues.

Likely cause is the power supply is underpowered for the rest of the system.

Second that. Not a software issue, but your power supply is challenged by two GPUs working at full steam.
There are not many softwares using the GPU as heavily as DR.

Hummmmm, even though it works in Mistaka fine? It's only DR that it crashes.

PSU is 1,300 watts. When the crashes happen it's not always rendering, it could be scrubbing the timeline with the GPUs at 10%.

It's only when using RAW files, timelines with all MP4 or MOV files never crash. Even if rendering and maxing the GPUs.

Well, you could try to switch the use of the GPUs off for BRAW and R3D under the Decode Options of DR's preferences. But that might slow you down even more than pulling one card.

Well, you could try to switch the use of the GPUs off for BRAW and R3D under the Decode Options of DR's preferences. But that might slow you down even more than pulling one card.

Yeah figured that would not be helpful.

BM support, is there any reason 2 GPUs will fail like this? Is it supported?

You might have a look at which rails of your power supply you have hooked your GPUs.
Maybe they are sharing the same rail and under full load the power supply will fluctuate.
Both cards together would draw up to 800 Watts.


Or it is a heat problem, the cards get very hot and with the typical top blower fans of gamer cards this might be a tipping point.

[Dwaine Maggart]

I agree it sounds like a power supply or power cabling issue. Reseat the cards. Reseat the power cables. Are you using proper power cables from the power supply to the cards? Y adapters could be a bad idea.

Or possibly a heat issue as Robert mentions. 2 3090's generate lots of heat under heavy load, and some cards don't exhaust out the back, but inside the case, which can cause issues.

2 GPUs should work fine in those workloads as far as Resolve is concerned.

No shared power cables or Y's, 1,300 watt PSU. I can mine crypto in downtime with both GPUs maxed for days and it's fine. I can run Mistika VR with both GPUs maxed for hours on end and it's fine. Case it large with lots of fans. GPU temps are 60-70C max.

It ONLY crashes with RAW files in Resolve. I can run Resolve and both GPUs just fine with MP4 or MOV files and it's fine. When it does crash from BRAW or RED RAW, it could be scrubbing the timeline, barely using any GPU power.

Is anyone running two 30 series GPUs and BRAW or RED Raw files?

Try different GPU drivers.
You are using RED Komodo files and this is different RAW format than old one (not a JPEG2000 anymore), so maybe this is the issues (but you said it happens on BRAW as well). It's still quite new territory. Maybe some specific combination in your machine which causes issue. Try disabling 1 card for a test.

When it happen on simple scrubbing then it's not power problem and anything load related. It will be matter of some driver, bug etc. issue.

Try different GPU drivers.
You are using RED Komodo files and this is different RAW format than old one (not a JPEG2000 anymore), so maybe this is the issues (but you said it happens on BRAW as well). It's still quite new territory. Maybe some specific combination in your machine which causes issue. Try disabling 1 card for a test.

When it happen one simple scrubbing then it's not power problem and anything load related. It will be matter of some driver, bug etc. issue.

I think it's a driver / software issue. Unplugged one card and it's stable. Running both cards, with RAW files, and it crashes.

And you're not running the GPUs in an SLI configuration?

Are they in appropriate PCIe x16 slots with 16 PCIe lanes in each?

And you're not running the GPUs in an SLI configuration?

Are they in appropriate PCIe x16 slots with 16 PCIe lanes in each?

Correct, no SLI, individual PCIe 4.0 slots. Motherboard is actually capable of running more GPUs.

My guess is it's the heat. 3090 is a 3 slot GPU. It produces a lot of heat. If you are not running a back-end blower type GPU, you must add an additional exhaust fan. Use a utility to track the CPU and GPU temps in real-time.

Heat from simple timeline scrubbing?
Not buying it
Sounds as a bug specific to given config.

I still think it would be worth testing with a 1600W power supply. You need overhead for current spikes. Having the computer shut off instantly under certain load is a clear sign. Even if rendering in other apps also utilizes both GPUs the curve of the power draw might be completely different in Resolve.

My guess is it's the heat. 3090 is a 3 slot GPU. It produces a lot of heat. If you are not running a back-end blower type GPU, you must add an additional exhaust fan. Use a utility to track the CPU and GPU temps in real-time.

Temps are fine, I can mine 24/7 in downtime without any heat issues. Never hit thermal throttle. It's a large case, 6 intake and 5 exhaust fans. GPUs have 3 fans each with a empty slot between them.

Heat from simple timeline scrubbing?
Not buying it
Sounds as a bug specific to given config.

My thought as well

I still think it would be worth testing with a 1600W power supply. You need overhead for current spikes. Having the computer shut off instantly under certain load is a clear sign. Even if rendering in other apps also utilizes both GPUs the curve of the power draw might be completely different in Resolve.

1,600 watt would require a larger outlet than I have available. But that would only explain a crash at render, not a crash simply scrubbing the timeline.

Seeing what support says.

Just to follow up.

Blackmagic support was very helpful. They looked through the log files and saw another display connected to my GPUs, even though I only have one display. Whatever was happening, the system thought there were two displays, one on each GPU, which they don't recommend.

Only thing I could think of was I have the Oculus Virtual Desktop Streamer app installed, maybe the system thought that was a display. Either way, I uninstalled that app, and just to make sure, moved my display connection to the other GPU. That way even if the system still thinks there's another display connected, they're both on the same GPU.

So far so good! BRAW files work fine with both GPUs running, I'll test some REDRAW shortly. Just a heads up in case this helps anyone else!

Tested a RED Weapon file from RED's site, no problems.

Only thing I could think of was I have the Oculus Virtual Desktop Streamer app installed, maybe the system thought that was a display. Either way, I uninstalled that app, and just to make sure, moved my display connection to the other GPU.

It must be this.

I've been fighting this issue for a while. Some short renders seemed fine, longer renders would crash and burn. Or simply playing the timeline with color grading enabled could crash it. Only in Resolve, nothing else would crash.

Seems like whatever my conflict was is tied to the noise reduction in Resolve. Disabling that would sometimes fix things.

I just gave up and unplugged one card to be able to work reliably. Today I had some time and ran the DDU app to uninstall drivers and all the random files created with the GPU driver installers. Found this video as a reference.

So far, with renders that would crash consistently with two cards in use, this has fixed it. I can render with both cards installed now after running this tool with the directions in the video.

I'm running the latest Resolve 18 Beta 3 currently. Just a follow up in case this helps anyone else.

Thanks for the update.
Do you know what kind of files were removed from the 'driver bloat remover'?
Which driver are you running now?

I usually install the geforce studio driver without geforce experience etc. Does that still come with unnecessary files?

Thanks for the update.
Do you know what kind of files were removed from the 'driver bloat remover'?
Which driver are you running now?

I usually install the geforce studio driver without geforce experience etc. Does that still come with unnecessary files?

The app tries to make it as if the drivers and apps where never installed. So back to square one without reinstalling windows.

I install GeForce Experience personally. Easy way to keep up to date on the studio drivers without searching the NVIDIA site. Doesn’t seem to hurt anything.

Previously when talking with support, they could see an extra monitor in the system logs, but I only had one attached display. While running the clean up app, one of the steps was removing any virtual displays. In my case that could have been the fix but I have no way of really knowing.

Could have been some stray files from an old driver installation or something.

I install GeForce Experience personally. Easy way to keep up to date on the studio drivers without searching the NVIDIA site. Doesn’t seem to hurt anything.

My experience as well.

Well the crashing with dual cards and BRAW is back, no clue why, so frustrating. Waiting on support to reply from the latest logs.

Editing MP4 media or exporting from Fusion works flawless. Only BRAW files crash at render or sometimes scrubbing the timeline.

Anyone else running dual 3080's or 3090's have this issue?

Looking back through the previous suggestions. Turned off "use GPU for Blackmagic RAW Decode" and so far that has fixed it. Funny enough, it's faster with that option turned off.

Maybe the threadripper has enough power for BRAW. The GPUs still ramp up so it's possible without doing the BRAW decode the GPUs have more resources left for other tasks, not sure. Gonna test some more.

Only (1) 3090 here, and I don't do gaming just Resolve editing and grading in 12k-8K but nevertheless, it felt more stable, maybe faster too when on a lark I switched to the Nvidia game driver instead of the Creators content driver. It occurred to me the game driver could get more frequent attention from Nvidia. I don't how Resolve leverages the creative driver if at all.

Only (1) 3090 here, and I don't do gaming just Resolve editing and grading in 12k-8K but nevertheless, it felt more stable, maybe faster too when on a lark I switched to the Nvidia game driver instead of the Creators content driver. It occurred to me the game driver could get more frequent attention from Nvidia. I don't how Resolve leverages the creative driver if at all.

I felt that the game driver was faster as well. When rendering it would show the FPS or the render and the game driver was consistently faster. Might be worth trying again for me since the studio driver has issues.

Turning off GPU for BRAW seems to have fixed the issue but I still got crashing on a larger project with lots of layers and graphics. Oh well.

Went to the Game Ready driver and saw a nice improvement in speed. On my big project that failed previously with the studio drivers, I got a lot further before it crashed.

Gonna keep testing.

Found someone on a Resolve Facebook group running a nearly identical computer setup to mine. Same motherboard, same dual 3090’s, same CPU. They had a 1,600W PSU instead of the 1,300W I was running. Since they weren’t experiencing crashes I ordered their same exact 1,600W PSU to test.

So far, it seems like the larger PSU has fixed the issue. Even though EVGA, who makes my 3090’s, says 1,300W is recommended for my setup. For those that guessed PSU, you were right!

Just posting as a follow up. Been rendering lots of proxies to prep for a new edit. All BRAW, all working so far.