ASUS x99-E WS user question and "GPU Memory Full"

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline
User avatar

Scott Stacy

  • Posts: 957
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

ASUS x99-E WS user question and "GPU Memory Full"

PostThu Mar 08, 2018 8:54 pm

The config manual lists one method for using 2 GPUs on this MB (use an expansion chassis). However, people use 2 Titan Xs on this MB all the time without significant incident. After 14.3 rolled out, I have been having "GPU Memory Full" problems, when in fact my GPUs have only been pushing 40% while monitoring performance during Smart Renders and when rendering out projects. I have even limited the number of fps in delivery renders so as to not push things too hard.

My current configuration, as directed by my IT support is Titan X (pascal) slot 1, Titan X (pascal) slot 3, Decklink 4k Pro slot 5, Areca RAID card slot 7. Preferences in GPU section are set to AUTO (CUDA). Both GPUs are selected and "Use GPU for Compute" is checked. I have tried it unchecked and have the same issue.

Windows 10, i7 6950x (10 core), 128 RAM.

Any thoughts?
Last edited by Scott Stacy on Sun Apr 22, 2018 3:44 pm, edited 1 time in total.
Scott Stacy, CSI
Colorist

Windows 10
HP Z8
RTX2080ti (x2)
Intel Xeon Gold 18 Core
128 RAM
NVME M.2 Samsung 970 2TB (x4)
Resolve 17.4
Offline
User avatar

Dwaine Maggart

Blackmagic Design

  • Posts: 11241
  • Joined: Wed Aug 22, 2012 2:53 pm

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostThu Mar 08, 2018 11:06 pm

If you are using very high resolution timelines, and multiple video tracks and/or lots of OFX/NR, and/or lots of parallel nodes, you could still exceed the 12GB of VRAM of the Titan X cards.

What sort of project is this happening in?
Dwaine Maggart
Blackmagic Design DaVinci Support
Offline
User avatar

Scott Stacy

  • Posts: 957
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 1:41 am

Hi Dwaine ... It is happening on a few timelines. Mostly on a 2k 2:39 black and white, 13 minute short film with around 7 nodes max, one parallel node with a couple windows. There are a handful of temporal noise reduction instances and some Beauty Box nodes. Funny, this never happened when the project was on my old 8 core D700 trashcan or on 14.1 on my PC. I have a simpler 4k DCI timeline, 8 minute short film. Maybe 5 instances of Temporal Noise Reduction, some keys. Nothing fancy. Also, on the 4k project timeline, sometimes I am getting a green screen until I hit play. This is random - even with no OFX or TRN.
Scott Stacy, CSI
Colorist

Windows 10
HP Z8
RTX2080ti (x2)
Intel Xeon Gold 18 Core
128 RAM
NVME M.2 Samsung 970 2TB (x4)
Resolve 17.4
Offline
User avatar

Dwaine Maggart

Blackmagic Design

  • Posts: 11241
  • Joined: Wed Aug 22, 2012 2:53 pm

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 1:52 am

Be aware that once you see that message, the GPU's are in a bad state, and will only recover with a Resolve restart. You shouldn't keep trying to do anything.

Is the message appearing on any particular clip? Or only when rendering? Or as soon as you load the project?

What NVIDIA driver are you using?
Dwaine Maggart
Blackmagic Design DaVinci Support
Offline
User avatar

Scott Stacy

  • Posts: 957
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 5:08 am

Yep ... I always restart. Usually, Resolve freezes up when I get that warning. I am on NVIDIA 390.65.

It used to occur all the time upon entering any project until Adam provided this code for Advanced Settings in preferences: Local.GPUPinnedHost.Enable = 0.

It occurs during smart renders, 1/4 of the way into a 10-12 minute film (not on any particular clip mind you or with any specific OFX).

Could this be a Resolve GPU memory management issue? For instance, I will start a smart render after working for a bit, the smart render will start and then as things progress I will get the notification. What I find baffling, is that in 14.1 and 14.2, I used to be able to run 2k timelines with multiple instances of TRN and even 20-30 instances of Beauty Box inside power windows plus Resolve OFX Lens Blurs on other clips and not ever get this warning.

Do you have any thoughts about this green screen issue and occasional static screen that shows up on the viewer, video output, and thumbnails? It goes away when I hit play. It's a random occurrence. I'm not seeing it on Alex, Red, or Ursa footage - just Panasonic Varicam LT 4K log footage. Anyway, this appears to be a 14.3 issues. However, I would rule out a hardware issue. Do you see a problem with the placement of my cards?

Thanks,

Dwaine
Scott Stacy, CSI
Colorist

Windows 10
HP Z8
RTX2080ti (x2)
Intel Xeon Gold 18 Core
128 RAM
NVME M.2 Samsung 970 2TB (x4)
Resolve 17.4
Offline
User avatar

Dwaine Maggart

Blackmagic Design

  • Posts: 11241
  • Joined: Wed Aug 22, 2012 2:53 pm

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 5:48 am

Looking at the manual for your mobo, it appears that slots 1 and 3 can be x16 slots.

But looking at the block diagram for your mobo, it shows slots 1 and 2 as x4 slots and all the rest as x8 slots. So that's a bit of a head scratcher.

You say this didn't happen with Resolve 14.1. Was that on this same mobo with this same GPU complement in the same slots?

If so, then this sort of smells like it could be driver related. Current driver is 391.01. You might give that a try. And if that behaves the same, try an older one, like 388.0

If changing drivers has no effect, go into Resolve Preferences, System - Hardware Configuration, set GPU Mode to CUDA, and GPU Selection Mode to Manual. Then deselect one of the Titan X cards, save the config, and restart Resolve. Then repeat the process and use the other Titan X. Is there any difference with one or the other? If so, this might show if one of the GPU's has an issue.
Dwaine Maggart
Blackmagic Design DaVinci Support
Offline

Chris Timson

  • Posts: 29
  • Joined: Thu Oct 12, 2017 3:05 pm

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 12:05 pm

I've had this on my Z840 (2x GTX1080)

It was a constant source of frustration and tried restarting the box each time I was about to play out a 45 min programme 1080/25. Every now and again I was still getting GPU mem full.

I eventually found a work around by instead of rendering at maximum speed, I turned it down to play out at maximum of 25fps. This seems to have solved it for now. Not ideal, but it does seem to play out reliably now.

Chris
i9-13900K - MSI MEG Z790 ACE - 64Gb - RTX3090 - BMD Mini Panel - 200Tb Thunderbolt 3 storage
Offline
User avatar

Scott Stacy

  • Posts: 957
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 7:14 pm

I checked the manual and found the Work Station block diagram for the x99-E WS and both list the slots as x16. I believe the non-work station mobo is the diagram you may have seen.

I have been on the same mobo and same GPU set up since 12.5. When the problem emerged around 14.3. My GPUs were in slots 3 and 5, as I had a TB3 card in 2. We thought it was the TB3 card causing problems and moved the GPUs up to 1 and 3 thinking it might be a resource issue, but that did not fix the problem. Is the not the best place for them to be from a resource management standpoint?

There seems to be an intermittent and context (Resolve OFX, Beauty Box, and TRN) issue going on. I rendered out a 8 minute Ultra HD timeline last night, but that footage did not have any TNR and minimal resolve lens blur. However, again, in the past, I could pile on OFX without this warning. I worry about doing any long form with this problem occurring.

I will try moving up to 391.01 and going back to 388.0. However, I think I already tried going back to 388.0. I even went back to an earlier system disk image. I will also try your hardware config experiment, too, and report back.
Scott Stacy, CSI
Colorist

Windows 10
HP Z8
RTX2080ti (x2)
Intel Xeon Gold 18 Core
128 RAM
NVME M.2 Samsung 970 2TB (x4)
Resolve 17.4
Offline
User avatar

Scott Stacy

  • Posts: 957
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostFri Mar 09, 2018 7:19 pm

Chris Timson wrote:I've had this on my Z840 (2x GTX1080)

It was a constant source of frustration and tried restarting the box each time I was about to play out a 45 min programme 1080/25. Every now and again I was still getting GPU mem full.

I eventually found a work around by instead of rendering at maximum speed, I turned it down to play out at maximum of 25fps. This seems to have solved it for now. Not ideal, but it does seem to play out reliably now.

Chris


I have my render speed typically set at 25-50 - especially, when there's heavy TNR and OFX on a timeline. It's discouraging to hear that your on a z840 and having to do this. I have always thought that z840s are pretty bullet proof. I was thinking about moving to a Z8. With 2 Titans, on my end, I would think it could handle more than 50fps, but I am on a i76950x 10 core system, which is not as robust as a z840.
Scott Stacy, CSI
Colorist

Windows 10
HP Z8
RTX2080ti (x2)
Intel Xeon Gold 18 Core
128 RAM
NVME M.2 Samsung 970 2TB (x4)
Resolve 17.4

Return to DaVinci Resolve

Who is online

Users browsing this forum: akyhne, anuctal, Bing [Bot], creal7, panos_mts, RedRider14, shikawkee, Steve Alexander and 173 guests