"GPU Memory Full" in Resolve Studio 15.2.4 Windows on 1080ti

This has crossed into the realm of insane. Here's what I'm talking about: https://photos.app.goo.gl/FbRTQcb7VZjLnAme9
Only 2m into rendering of my UHD project with H.265 output, with only 2 or 3 UHD clips rendered before the failure, and the rest being FHD clips shot on a phone, with GPU Shark reporting barely 20% GPU usage of my 11GB 1080ti, I get the dreaded "Your GPU memory is full". My timeline resolution is already at 1080. There's NO WAY this is legitimate unavoidable issue - this is clearly a bug in DV Resolve Studio. System memory usage was around 13 GB out of 24 GB. We, the users, need to make our voices heard somehow. This is ridiculous.

Update:
I tried plugging monitor to both 1060 and 1080ti, and use UI cards for UI only and for compute too, with no difference at all - processing fails at exactly the same point no matter what. GPU Shark reports GPU memory consumption as low as 8% sometimes, when failure occurs.
This is very likely not an actual lack of memory situation. It's a bug somewhere.

Work-around Update:
I had to disable GTX 1060 in Windows Device Manager, where I kept my monitor connected, and use GTX 1080ti for both UI and Compute. Clearly, there's a bug somewhere, because reducing overall amount of GPU memory from total 17GB on both cards to 11GB, made "GPU Memory is full" error go away. NVENC still produces videos that get stuck during the playback, but that's a different issue.

This has crossed into the real of insane. Here's what I'm talking about: https://photos.app.goo.gl/FbRTQcb7VZjLnAme9
Only 1m30s into rendering of my UHD project with H.265 output, with only 2 or 3 UHD clips rendered before the failure, and the rest being FHD clips shot on a phone, with GPU Shark reporting barely 20% GPU usage of my 11GB 1080ti, I get the dreaded "Your GPU memory is full". My timeline resolution is already at 1080. There's NO WAY this is legitimate unavoidable issue - this is clearly a bug in DV Resolve Studio. System memory usage was around 13 GB out of 24 GB. We, the users, need to make our voices heard somehow. This is ridiculous.

Always had the same issue on Windows but switched to Manjaro Linux about 3 months ago and never had this issue again working on my 4K projects on a 1080Ti. Can even add neat video at highest quality setting.

It's either Windows or Resolve and it's optimizations on Windows.
Even had it while just playing back, rendering using NVenc was completely impossible.

This has crossed into the real of insane. Here's what I'm talking about: https://photos.app.goo.gl/FbRTQcb7VZjLnAme9
Only 1m30s into rendering of my UHD project with H.265 output, with only 2 or 3 UHD clips rendered before the failure, and the rest being FHD clips shot on a phone, with GPU Shark reporting barely 20% GPU usage of my 11GB 1080ti, I get the dreaded "Your GPU memory is full". My timeline resolution is already at 1080. There's NO WAY this is legitimate unavoidable issue - this is clearly a bug in DV Resolve Studio. System memory usage was around 13 GB out of 24 GB. We, the users, need to make our voices heard somehow. This is ridiculous.

How many effects and how much noise reduction?

This also occurs occasionally with my Titan xp (12GB) on Windows 10. It definitively gets worse when using noise reduction, sharpening etc. together with H265.

if you need export, export DPX, first then encode from there: if the machine freeze, you don't lose anything.

It's either Windows or Resolve and it's optimizations on Windows.
Even had it while just playing back, rendering using NVenc was completely impossible.

Or NVidia driver/encode API..
I had no trouble with the workaround: DR (slow, unaccelerated rendering) => DNxHQ, then fast ffmpeg+nvenc => H.265. But it's time consuming, awkward as the workflow, and requires dual loss-incurring compression. I'm unlikely to move to Linux as this Windows system is my multi-purpose computer and I can't dedicate it to just video work. The product of such terrible quality should not have be en released or should be promptly be fixed.

How many effects and how much noise reduction?

No NR at all on scenes/clips all the way to the failure point. The rest is basic color grading and stabilization.

I am certain this is not a legitimate out of memory situation, as I hit this failure when GPU Shark showing as little as 8% GPU memory utilization. Thus it has nothing to do with actual memory amount or how much of it is consumed. It could be a Windows handle leak in a C/C++ program, given how consistently it fails at the very same place (probably the same frame count). I tried tree different NVidia driver versions: latest 418.91, the one on the screenshot, and 399.xx - the one someone else used to work around this problem, but that didn't help me. It's sad to watch how BMD is now following the downward quality trajectory of most other NLE products, which initially it seemed to be able to defy.

This also occurs occasionally with my Titan xp (12GB) on Windows 10. It definitively gets worse when using noise reduction, sharpening etc. together with H265.

I wish I was so lucky to have this happening occasionally. It fails at precisely the same point after rendering just a handful of clips in the timeline, with most scenes before failure being FHD clips recorded on a phone, with no NR at all. That's why this is beyond ridiculous.

if you need export, export DPX, first then encode from there: if the machine freeze, you don't lose anything.

Thanks. I have a pre-existing work-around with DR=>DNxHQ => ffmpg+nvenc => H.265, but I was really looking forward to streamlining my process by eliminating the slow DNxHQ step.

NVenc still feels like beta software(same for the AMD hardware encoder), it's fast when it works, but it doesn't always work (and the file size is also a lot bigger than software x265).

We used to Cineform everything and then StaxRip it to x264 or x265 (NVenc or AMD hardware encoding when speed was important and software when filesize and quality was important). Since BRAW a lot have changed for us.

NVenc still feels like beta software(same for the AMD hardware encoder), it's fast when it works, but it doesn't always work (and the file size is also a lot bigger than software x265).

We used to Cineform everything and then StaxRip it to x264 or x265 (NVenc or AMD hardware encoding when speed was important and software when filesize and quality was important). Since BRAW a lot have changed for us.

NVEnc lossless is what I want out of Resolve as master. Much faster than Cineform to render, a fraction of the space for things with little motion. It drops to 3 MBit for 4k 60p for a static image. I need to do some screencasts and tutorials lately and rendering out Terrabytes of lossy Cineform for nothing is such a pain on a laptop. Also it plays back easier than Cineform in VLC.

I would try turning off image stabilization on all the clips and see if that makes a difference.

NVEnc lossless is what I want out of Resolve as master. Much faster than Cineform to render, a fraction of the space for things with little motion. It drops to 3 MBit for 4k 60p for a static image. I need to do some screencasts and tutorials lately and rendering out Terrabytes of lossy Cineform for nothing is such a pain on a laptop. Also it plays back easier than Cineform in VLC.

10 bits NVenc lossless is not good enough for our commercials (news items are no problem since it's only 8 bits output). Cineform Filmscan 1,2 and 3 is a lot better quality. When ever possible, we like the source material to be atleast 2 bits higher than our delivery material. BRAW Q0 is 16 bits logged to 12 bits.

Also our GTX970's don't support NVenc 10 bits lossless, you need a atleast a pascal card for that.

Cineform also runs a lot better than HVEC on computers without hardware accelerated decoding.

NVEnc lossless is what I want out of Resolve as master. Much faster than Cineform to render, a fraction of the space for things with little motion. It drops to 3 MBit for 4k 60p for a static image. I need to do some screencasts and tutorials lately and rendering out Terrabytes of lossy Cineform for nothing is such a pain on a laptop. Also it plays back easier than Cineform in VLC.

10 bits NVenc lossless is not good enough for our commercials (news items are no problem since it's only 8 bits output). Cineform Filmscan 1,2 and 3 is a lot better quality. When ever possible, we like the source material to be atleast 2 bits higher than our delivery material. BRAW Q0 is 16 bits logged to 12 bits.

Also our GTX970's don't support NVenc 10 bits lossless, you need a atleast a pascal card for that.

Cineform also runs a lot better than HVEC on computers without hardware accelerated decoding.

HEVC lossless currently makes no sense. It generates larger files while being more complex to encode and decode than H264 lossless.

On my system lossless H264 is easier to decode than Cineform 10 bit. Never compared it to HVEC lossless, though.

If you need more than 10bit then of course it's not useful for your case. It would speed up my workflow tremendously.

GTX1060 6GB + GTX1080Ti 11GB both used = 6GB of vram available, not 11GB.

If your system monitor is plugged to the 1060 you will get 4GB free, Windows 10 uses about 2GB of vram.
Don’t assign the GTX1060 to DaVinci Studio.

Regards

Well spotted Gabriele. That info isn't in the post but in the screenshot.

Just noticed that after installing Manjaro Linux on my older laptop (NVIDIA 980M 4GB Vram) I can use Davinci Resolve Studio editing and even grading 4K DCI timelines as well - something that has been almost impossible on Windows 10 running on the same machine.

WTH is wrong with Windows 10 or the Windows version?

NVEnc lossless is what I want out of Resolve as master. Much faster than Cineform to render, a fraction of the space for things with little motion. It drops to 3 MBit for 4k 60p for a static image. I need to do some screencasts and tutorials lately and rendering out Terrabytes of lossy Cineform for nothing is such a pain on a laptop. Also it plays back easier than Cineform in VLC.

10 bits NVenc lossless is not good enough for our commercials (news items are no problem since it's only 8 bits output). Cineform Filmscan 1,2 and 3 is a lot better quality. When ever possible, we like the source material to be atleast 2 bits higher than our delivery material. BRAW Q0 is 16 bits logged to 12 bits.

Also our GTX970's don't support NVenc 10 bits lossless, you need a atleast a pascal card for that.

Cineform also runs a lot better than HVEC on computers without hardware accelerated decoding.

Please try DNXHR as well.
I had some banding issues even using the filmscan option - which should be 10 bit at least. DNXHR HQX is visually lossless to me.
Also - nvenc on my 1080Ti is garbage lol

NVEnc lossless is what I want out of Resolve as master. Much faster than Cineform to render, a fraction of the space for things with little motion. It drops to 3 MBit for 4k 60p for a static image. I need to do some screencasts and tutorials lately and rendering out Terrabytes of lossy Cineform for nothing is such a pain on a laptop. Also it plays back easier than Cineform in VLC.

10 bits NVenc lossless is not good enough for our commercials (news items are no problem since it's only 8 bits output). Cineform Filmscan 1,2 and 3 is a lot better quality. When ever possible, we like the source material to be atleast 2 bits higher than our delivery material. BRAW Q0 is 16 bits logged to 12 bits.

Also our GTX970's don't support NVenc 10 bits lossless, you need a atleast a pascal card for that.

Cineform also runs a lot better than HVEC on computers without hardware accelerated decoding.

Please try DNXHR as well.
I had some banding issues even using the filmscan option - which should be 10 bit at least. DNXHR HQX is visually lossless to me.
Also - nvenc on my 1080Ti is garbage lol

How can a super fast lossless encode be garbage?

How can a super fast lossless encode be garbage?

It looks as bad as fast as it encodes. Does it answer the question?
80Mbit H265 Nvenc encodes basically look worse (almost no texture, grain is gone, banding etc) than a 7Mbit x265 10bit encode using handbrake after DNXHR HQX - both 4K DCI Scope. lol

GTX1060 6GB + GTX1080Ti 11GB both used = 6GB of vram available, not 11GB.

If your system monitor is plugged to the 1060 you will get 4GB free, Windows 10 uses about 2GB of vram.
Don’t assign the GTX1060 to DaVinci Studio.

To clarify, I have my monitor connected to 1060 - I just verified it in the Device Manager, and DR sees it the same way when I turn off "Use display GPU for compute". I tried setting DR to do both w 1060: for UI only, and use it for compute, with no difference in outcome whatsoever.

I think you keep treating this as a legit lack of memory situation, while GPU Shark reports memory usage on both cards anywhere from 8% to 50% and everywhere in between when the failure occurs. Can you please incorporate some explanation of why GPU Shark memory usage report is not supporting lack of memory thesis?

Many thanks,
Vlad.

on this planet there are few capable good programmers on Cuda.
Neither Adobe has good Cuda coders.

How can a super fast lossless encode be garbage?

It looks as bad as fast as it encodes. Does it answer the question?
80Mbit H265 Nvenc encodes basically look worse (almost no texture, grain is gone, banding etc) than a 7Mbit x265 10bit encode using handbrake after DNXHR HQX - both 4K DCI Scope. lol

I think you don't understand what lossless is. It's bit identical to what you feed into it, which you can verify with FFMPEG. That mode works perfectly and extremely fast.

Obviously x264/x265 provides much better quality than GPU based encoders in lossy mode, that's why everyone exports to Cineform or DNx and encodes from there with FFMPEG for final deliveries. Still the lossy modes are very useful to quickly render previews out with small file size. It's just not the thing you use for final deliveries.

GTX1060 6GB + GTX1080Ti 11GB both used = 6GB of vram available, not 11GB.

If your system monitor is plugged to the 1060 you will get 4GB free, Windows 10 uses about 2GB of vram.
Don’t assign the GTX1060 to DaVinci Studio.

To clarify, I have my monitor connected to 1060 - I just verified it in the Device Manager, and DR sees it the same way when I turn off "Use display GPU for compute". I tried setting DR to do both w 1060: for UI only, and use it for compute, with no difference in outcome whatsoever.

I think you keep treating this as a legit lack of memory situation, while GPU Shark reports memory usage on both cards anywhere from 8% to 50% and everywhere in between when the failure occurs. Can you please incorporate some explanation of why GPU Shark memory usage report is not supporting lack of memory thesis?

Many thanks,
Vlad.

I'd disable the 1060 and use the 1080 for compute and display. Else every computed frame needs to travel from the 1080 to 1060 to actually display it. There might of course still be issues with memory management, though.

20% GPU memory utilisation is shockingly small, and the math for the 1080 doesn't add up. The GL report says 3178 of 11264 used (28%), but later GPU memory is reported as 20.3%

Out of interest what do the GPU graphs look like in the Windows Task Manager > Performance tab?

I only have a single 1060 and the math in GPU Shark does add-up, and I can easily see 95+ % memory utilisation -- before DR semi-crashes with GPU memory issues.

As the timeline is partially cached, enabling 'use render cached images' in Advanced delivery settings may help to get you through this render.

Also... all of your desktop applications appear to be using the 1080 and not the 1060. So there seems to be little advantage in having it installed. Disabling your 1060 (as suggested) in Hardware Manager and restarting Resolve - should give you more accurate utilisation reporting, and potentially better performance - as BMD advise against using multiple non-identical GPU's.

20% GPU memory utilisation is shockingly small, and the math for the 1080 doesn't add up. The GL report says 3178 of 11264 used (28%), but later GPU memory is reported as 20.3%

Out of interest what do the GPU graphs look like in the Windows Task Manager > Performance tab?

I only have a single 1060 and the math in GPU Shark does add-up, and I can easily see 95+ % memory utilisation -- before DR semi-crashes with GPU memory issues.

As the timeline is partially cached, enabling 'use render cached images' in Advanced delivery settings may help to get you through this render.

Also... all of your desktop applications appear to be using the 1080 and not the 1060. So there seems to be little advantage in having it installed. Disabling your 1060 (as suggested) in Hardware Manager and restarting Resolve - should give you more accurate utilisation reporting, and potentially better performance - as BMD advise against using multiple non-identical GPU's.

Thank you for an insightful reply. I started using Task Manager and I can see that GPU Shark seems to have a bug in showing how much GPU memory is being used: it shows memory usage of 1060 for in both 1080ti and 1060 columns. Also, GPU processor utilization displayed is not identical between the GPU Shark and Task Manager. Task Manager shows more plausible numbers, like 60% varying at +/-10% while working.

I have also made another screenshot a few seconds after the latest failure, showing Task Manager 1080ti graph, GPU Shark, and Device Manager, where it shows my monitor connected to the 1060:
https://photos.app.goo.gl/kQWs9mgbF6jkuK4ZA
One can also see that failure occurs exactly at the start of a simple Cross-Dissolve transition between two UHD/60 clips shot on different cameras.
Now for the kicker, my previous attempt to render has run through successfully, albeit one clip ended up rendered as frozen in the final movie. I re-ran rendering again, and it failed again at exatly the same point where it failed before.

Could you please clarify how you have concluded that apps are using 1080ti and not 1060? I watched 1060 utilization too, and it appears to be consistent with 1060 being used only for UI, as set in DR. All 1080ti metrics go to 0% utilization as soon as I exit from DR. 1060 shows no change in utilization of anything with DR or without DR running. With DR on, 1080ti utilization goes up after DR starts.

I tried to follow a suggestion of disabling 1060 in Device Manager and moving UI and Compute to 1080ti. With that rendering passed the usual point of failure, but got stuck at about 57% of the timeline with no error message. I will try a couple more runs in this configuration later, and will post the update.

Still, actual memory utilization does not appear to be a factor. I am still pretty convinced that it's bug, or bugs, in either DR, or NVENC, or both, that cause the failure, and not lack of resources.

Thank you again,
Vlad.

You can deactivate Cuda for your selected gpu from the Nvidia control panel, you haven’t exclude it from the hardware list. DaVinci will never see your GTX1060 as Cuda capable but will try automatically to set both gpus in OpenCL.
Definitely, as Chaos Group does for Vray RT, DaVinci needs an external app that configures Cuda or OpenCL gpus before to start the Resolve program otherwise many customers with different brands or gpu models will have issues.
Auto recognition of gpus and api doesn’t work for all combinations.

Regards

Glad to see you have made some progress. Mystical or otherwise.

My conclusion about the 1060 non-utilisation was based on the allocation of the '3D apps' like chrome etc. which appeared to be listed under the 1080. But I now believe that that list may just be a single column list, not related to any particular card. So my observation is respectfully withdrawn.

When you disable the 1060 - it will not show up as GPU1, and will not be an option in DR. Looks like you may have been working in Device Manager under System Devices. You should however disable the 1060 under "Display adapters". I have my Intel 630 disabled.

Display Adapters

Disable 1060 Capture.JPG (32.54 KiB) Viewed 5986 times

Did you also try using the Use Render Cache in Advanced Settings in the Delivery Page? I say this because the troublesome Cross-Dissolve transition appears to be render cached.

And yes - seems like something buggy is going on.

I followed the suggestion to disable one of two cards, the 1060, and that helped: I no longer get the out of memory error, and renderring completes. I'm still getting output that hangs players, but that's a different issue for a different thread. Nvidia and BMD have a lot of work remaining on their products though.
Thanks, everyone.

Vlad