Sick and tired of "GPU Memory is Full"

I have a 2 hours concert footage shot in a low light. I made all the color grading then added NR and tried to render it. Got "GPU MiF" error.

ok, I made color grading first and got the intermediate result. Now I try to apply just NR and I constantly get "GPU memory is full"! This is with 10 core processor, 16 Gb of RAM and Nvidia 1080ti GPU (Windows 10)!


Then I tried to make this NR processing piece by piece - sliced the timeline by 10 min. And producing single clips. NO! You can't do that either!

Tell me, what ELSE I can do to apply Noise Reduction? Spend $6500 on a Quadro card??

v.14.3.1

If you have lots of noise reduction to do I would render out the whole thing to a master file (Prores, DNxHR, DPX sequence, whatever fits your workflow) and denoise that. Noise reduction will sap a bunch of processing power and depending on the footage/other corrections your memory could be eaten up.

ok, I made color grading first and got the intermediate result. Now I try to apply just NR and I constantly get "GPU memory is full"! This is with 10 core processor, 16 Gb of RAM and Nvidia 1080ti GPU (Windows 10)!

I wonder if the out of GPU memory message could be an indirect consequence of having a very low (16GB) amount of system memory.

Yeah, this error does sound like wonder... And I wonder if developers even know what does it mean.


If you do know you have a problem with such a processing (NR), please provide throttling, settings that would clearly make it work, anything!


My source is 10 bit DNxHR. What else can I do with it?

My source is 10 bit DNxHR. What else can I do with it?

what format source: HD? UHD? more ?

My source is 10 bit DNxHR. What else can I do with it?

what format source: HD? UHD? more ?

UHD

UHD + corrections + temporal NR will eat a lot of memory.

UHD + corrections + temporal NR will eat a lot of memory.

Exact ! +100

ok, thank you. Well, if that's 100% positive I'm upgrading to 128 Gb of RAM.
But what if this doesn't help...

ok, thank you. Well, if that's 100% positive I'm upgrading to 128 Gb of RAM.
But what if this doesn't help...

Read my first post in this thread.

If you have lots of noise reduction to do I would render out the whole thing to a master file (Prores, DNxHR, DPX sequence, whatever fits your workflow) and denoise that. Noise reduction will sap a bunch of processing power and depending on the footage/other corrections your memory could be eaten up.

You didn't say if that's RAM or GPU memory.

No I didn't but if you render it out first and denoise the master, then instead of Resolve having to use processing power for the corrections and NR it can save up the system resources to work on the NR only.

No I didn't but if you render it out first and denoise the master, then instead of Resolve having to use processing power for the corrections and NR it can save up the system resources to work on the NR only.

So I did exactly opposite. I graded everything first. And now trying just to apply NR. And it throws this error.

I just want to know exactly which part of my machine to upgrade. If not everything which would be a non-sense.

Yes you should grade first and then export a new file from your timeline as a master file that has your grades in it. Then bring the graded file back in Resolve and apply the NR to that new rendered file. I don't know how many times I can write the same directions in a different way so if you still don't get it then please ask for more information.

No I didn't but if you render it out first and denoise the master, then instead of Resolve having to use processing power for the corrections and NR it can save up the system resources to work on the NR only.

So I did exactly opposite. I graded everything first. And now trying just to apply NR. And it throws this error.

I just want to know exactly which part of my machine to upgrade. If not everything which would be a non-sense.

Hello Alex,

Here I come with 32 GB RAM (CAS 14) to make TNT 5 in UHD and just an intel 6850k (6 cores + 6 hyperthreading) + 2 gpus gtx 1070 but what matters is not having a bottleneck. (It's a resource balance, including the PSU, the HDD and of course: the settings ...Input Source from a HDD, output delivery to others HDD(s)...)

You have a good Nvidia 1080i graphics card: I think having 32 or 64 GB Ram will be a good start.

So insufficient: push a little more RAM but if evolutions: make sure that any purchase is 'scalable' and not to throw for something else.

But what works for some is not 100% certain in others. (must have the same hardware ...and settings)

Yes you should grade first and then export a new file from your timeline as a master file that has your grades in it. Then bring the graded file back in Resolve and apply the NR to that new rendered file. I don't know how many times I can write the same directions in a different way so if you still don't get it then please ask for more information.

What are we agruing about (if we do )? That's then exactly what I did.

So you're positive I need a RAM upgrade? Just asking...

Thank you.

Jean Claude, thank you! I'll try to have a RAM upgrade and then see what else can be improved.

Upgrade to 32 GB and get a VEGA FE with 16 GB, when the 16GB is full it will swap the less needed parts to the system memory. It also has 10 bits color overlay in OpenCL. Doing more and more 8k workflows these days and have never run out of memory with the VEGA FE's(using them for a year now).

New price $999

and get a VEGA FE with 16 GB,

Hi.

There are two different versions of AMD VEGA Frontier Edition (FE) Graphics cards with 16 GB.
One is Air cooled and the other is Liquid cooled. I expect that the Liquid cooled version is best for use with Resolve?
Here is a link:

https://www.ebay.com/itm/AMD-Radeon-Veg ... Sw2ytbnAvL

Regards Carsten.

Thanks, guys! I'm aware of that exact card. But I'm going to up my RAM first. 1080ti should be enough I guess.

But BMD developers should have ways to throttle the NR processing!

The advantage of VEGA 8 and 16GB version is that it can swap to system memory. NVidia cards don't have this feature. But it's always good to have more system memory.

ok, I found settings which do not blow up Resolve. But.. it's been running for 3 hours and then Resolve dies

Oh, nice. It doesn't start anymore. Even after reboot. Sigh...

Oh, nice. It doesn't start anymore. Even after reboot. Sigh...

What version ar you running? 15.1.1?

ok, I found settings which do not blow up Resolve. But.. it's been running for 3 hours and then Resolve dies

What changes did you make?
Did you monitor the temperatures of both the CPU and GPU during rendering?

I specified version in the very first post: v.14.3.1

I change NR settings to use 1 frame (for both NR) and choose Faster with smaller radius. Then selected exporting to 10 bit DNxHR. The drive it outputs is Samsung 970 PRO NVMe SSD 1Tb.

I just reinstalled the app. Let's try it again. If this time it doesn't finish I'll split it into 2 pieces (1 hour each). But this is nuts.

I specified version in the very first post: v.14.3.1

I change NR settings to use 1 frame (for both NR) and choose Faster with smaller radius. Then selected exporting to 10 bit DNxHR. The drive it outputs is Samsung 970 PRO NVMe SSD 1Tb.

I just reinstalled the app. Let's try it again. If this time it doesn't finish I'll split it into 2 pieces (1 hour each). But this is nuts.

Do you have to do it all again? You should have a file up to where it crashed, try if you can use that. If you can use it then make sure you overlap a bunch of frames at the end and render the remainder.

Next time, I deleted the file... Well, too fast.

Now, after the reinstall and reboot my project can't be open anymore. Resolve just stuck in the processes doing nothing.

ok, let's create a new project then, shall we?

It's been 16 hours of no luck. Resolve just gets annihilated after couple of hours of processing.

So I split it into pieces 30 min each. Now, see if we can get *something*!

FWIW:


I used to run Resolve 14 on 32Gb RAM with a 4-core 6800k and a single 980ti and graded and rendered several projects in UHD and 4K DCI. Now I run Resolve 15 on 32Gb RAM with a 6-core 8700k and two 1080ti's, also no problems with 4k renders and a bunch of NR etc. (except my renders are now much much faster).

The point is that I have never encountered the GPU Memory Full error with either one of those card combinations or CPU's…but the one constant has been the 32Gb of RAM…so maybe you need more system RAM and Resolve is giving you a misleading error.

Also, make sure your Nvidia drivers are up to date.

Also, make sure your Nvidia drivers are up to date.

+1

Hello Alex,
and with up to date driver Attempt to do a 'Custom' installation and ask to do a Clean installation ?

Drivers are up to date guys.

OMG, I just passed first 30 min! Do not breath everyone! )

GPU memory is full means that you need a graphic card with more RAM. to my understanding this has nothing to do with system ram. the more gpu ram the better. 12gb and you are all in, 8gb you can get into troubles with UHD+ and NR.

So, results are out of reach because my 1080ti has 11Gb? I'm doomed.

I too am running into "GPU Memory is Full" problems. (I've got a GTX Titan GPU and a Titan Black GPU installed). I haven't been doing noise reduction yet but I do have a fair bit going on in this grade even though the timeline is only a couple of minutes long.

I would have thought that you could essentially string out an infinite series of nodes and that it would just take longer to render the project or take the time to display a particular frame such as in After Effects where you can just keep on adding layer after layer.

Does Resolve work in a fundamentally different way then other applications where you can only do as much as your hardware will allow or are we running into a bug here? If not a bug how the heck can you grade a feature film length project when a 2 minute project runs into hard limits?

Finally! I got all my 4 30-minutes pieces rendered (with minimal NR settings). But then I checked the sound... The last piece has a periodic click (level drop, I don't know how to explain). So I started to render the last piece again...

The thriller continues...

Hey Alex, are you working in Dual Screen mode?

I am now mucking about my project that is giving me the Mem full errors and it seems that if I toggle from Dual Screen mode to Single screen mode I don't get the mem full notice. I've switched back to Dual Screen now and started adding nodes and it seems to be working.

I suspect there is some bug issues going on here...

No, I'm using single UHD monitor.

Finally! I got all my 4 30-minutes pieces rendered (with minimal NR settings). But then I checked the sound... The last piece has a periodic click (level drop, I don't know how to explain). So I started to render the last piece again...

The thriller continues...

You might consider rendering as a DPX frame sequence, that way if the render crashes, you can pick up where you left off. Not ideal, but it will eventually get you to the finish line.

As far as GPU memory, the 1080ti is plenty for 4K, hell the 980ti with almost half that is also plenty. I even remember doing some 4K stuff with my 970 which had 2Gb of RAM….crazy!

So no, you don't need a Titan X, you don't need new/more GPU's, you should be fine. I think something else is going on which is causing problems. But I also think you're spinning your wheels here in the forums. You should be contacting BMD support directly.

my current theory is there is a bug in the display/GPU allocations. You might try resizing your display screen resolution or something to get Resolve to reset the GPU stuff.

I was able to continue adding nodes with Noise reduction in them and not get the GPU full warning yet.

Sounds like there are bugs with allocation for some people. I never run out of memory on 4K timeline resolution with 1080 Ti. Very easy with 8K. Using two UHD monitors with dual screen mode.

Does Resolve work in a fundamentally different way then other applications where you can only do as much as your hardware will allow or are we running into a bug here? If not a bug how the heck can you grade a feature film length project when a 2 minute project runs into hard limits?

Due to the way Resolve works, a 10 second timeline will consume the same amount of GPU memory as a 10 hour timeline. For spatial filtering like colour correction nodes, one frame would take up however many times its size in memory as you have nodes. For temporal filtering however (denoise, motion blur) there is a compound effect where multiple frames have to be loaded into GPU memory in order to process them. This is where you run into "GPU Memory is Full" messages.

The reason for this way of doing things is that shuttling data between GPU and system memory is very expensive on the system bus and keeping data on the GPU ensures that realtime playback is even possible at all with a graded timeline. I do agree that a more flexible "I don't care how slow things render" fallback mode for heavy effects should be implemented, where excess data is swapped out to system memory, maybe even to disk if necessary, so that processing never halts.

For spatial filtering like colour correction nodes, one frame would take up however many times its size in memory as you have nodes.

This is a brute force approach with no memory management whatsoever. I (want to) believe Resolve does things in more elegant way than this. Whichever way node graph evaluation is done, recursively (each node calls its inputs and then processes itself) or by preprocessing the graph (flattening it to order of operations), each node only needs the result if its inputs to do its job, node results that have been processed and "consumed" can be discarded from memory if this room is needed for something else.

Does Resolve work in a fundamentally different way then other applications where you can only do as much as your hardware will allow or are we running into a bug here? If not a bug how the heck can you grade a feature film length project when a 2 minute project runs into hard limits?

The reason for this way of doing things is that shuttling data between GPU and system memory is very expensive on the system bus and keeping data on the GPU ensures that realtime playback is even possible at all with a graded timeline.

The irony of this logic is that realtime playback is worth nothing if you get a memory full error, isn't it? Running out of memory is not a preformance enhancement feature, it is bad memory management. What is usually done in such cases is that every node is weighted by its processing weight and stuff that is heavy gets priority in cache and lighter stuff is discarded. In case of branching and merging, each branch can be processed to last node and only that is kept in memory until it is time to do the merging. Keeping stuff in memory while you can't even complete processing is not helping with getting things done. There should always be a fallback mechanism where necessary data is moved to system ram and pulled from there if it is time to use it. Yes, it is slow, but would you choose playing/rendering slower or not doing it at all due to errors?

Do not export a mov, if it crash, you lose it all.

Export dpx 10 bits, in case it fail, you can safely restart from the second to last shot you where.

Then you do the mov from that.

Be sure to export an EDL before you render out your timeline sans NR. Then import it as Pre-conformed EDL to cut it up. Every scene may need different NR settings. Remember, there will be better NR solutions in the future—Having a version of your master without is a good idea (also will prevent any future rejections due to NR artifacts).

Also, I have had GPU full memory warnings for some reason when the render disk hasn't kept up. Rendering to a proper RAID has fixed this. Try reducing render speed too.

on a side note, all the tutorials ive been watching have said to use NR first, then CC.
However, I read hear that users are doing it the other way around.

I'm guessing, then, that this is more of a personal preference thing - either NR then CC, or CC then NR.
Personally, it makes more sense to me to remove noise before CC. however, this implies that CCing doesn't add its own noise that then needs to be removed.

i tend not to use NR much, when i do, i'll grade first, and then add NR at the begining of the node tree once i know how little i can get away with

on a side note, all the tutorials ive been watching have said to use NR first, then CC.
However, I read hear that users are doing it the other way around.

I'm guessing, then, that this is more of a personal preference thing - either NR then CC, or CC then NR.

It's more of a technical ignorance thing than a personal preference thing. Noise is generated at the sensor, and its distribution is linear in nature, so it should be removed in linear space, preferably even before white balance correction. Any colour corrections will change the distribution of the noise making it harder to remove. Additionally, qualifiers are greatly affected by noise and you will get a better, more natural grade if you denoise before adding nodes with qualifiers.

I'm on a 2014 iMac and I get that message when my machine's hard drive (not the RAM) gets more than 75 percent full. It seems to have nothing to do with other apps or shortage of RAM. I clear out the hard drive, reboot and I'm back in business.

Be sure to export an EDL before you render out your timeline sans NR. Then import it as Pre-conformed EDL to cut it up. Every scene may need different NR settings. Remember, there will be better NR solutions in the future—Having a version of your master without is a good idea (also will prevent any future rejections due to NR artifacts). Also, I have had GPU full memory warnings for some reason when the render disk hasn't kept up. Rendering to a proper RAID has fixed this. Try reducing render speed too.

John is very wise. This would have been my exact advice: drop the render speed to 10fps or 5fps, use a pre-colored clip, and cut it up with an EDL.

We have been just now encountering some render failures in 15.1 close to the end of a 2-hour file, but it's intermittent and unpredictable. Most of the time, it seems to work. NR was not a factor.

You can make a case for Neat Video vs. Resolve NR: Neat is costly ($295) and slow (2fps tops on my machine), but it is cleaner. The new Resolve Enhanced SNR is pretty amazing and comes very close to the same results if you do a luma/chroma split at different ratios, but it's also very, very slow.