Page 1 of 1

ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Thu Mar 08, 2018 8:54 pm
by Scott Stacy
The config manual lists one method for using 2 GPUs on this MB (use an expansion chassis). However, people use 2 Titan Xs on this MB all the time without significant incident. After 14.3 rolled out, I have been having "GPU Memory Full" problems, when in fact my GPUs have only been pushing 40% while monitoring performance during Smart Renders and when rendering out projects. I have even limited the number of fps in delivery renders so as to not push things too hard.

My current configuration, as directed by my IT support is Titan X (pascal) slot 1, Titan X (pascal) slot 3, Decklink 4k Pro slot 5, Areca RAID card slot 7. Preferences in GPU section are set to AUTO (CUDA). Both GPUs are selected and "Use GPU for Compute" is checked. I have tried it unchecked and have the same issue.

Windows 10, i7 6950x (10 core), 128 RAM.

Any thoughts?

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Thu Mar 08, 2018 11:06 pm
by Dwaine Maggart
If you are using very high resolution timelines, and multiple video tracks and/or lots of OFX/NR, and/or lots of parallel nodes, you could still exceed the 12GB of VRAM of the Titan X cards.

What sort of project is this happening in?

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 1:41 am
by Scott Stacy
Hi Dwaine ... It is happening on a few timelines. Mostly on a 2k 2:39 black and white, 13 minute short film with around 7 nodes max, one parallel node with a couple windows. There are a handful of temporal noise reduction instances and some Beauty Box nodes. Funny, this never happened when the project was on my old 8 core D700 trashcan or on 14.1 on my PC. I have a simpler 4k DCI timeline, 8 minute short film. Maybe 5 instances of Temporal Noise Reduction, some keys. Nothing fancy. Also, on the 4k project timeline, sometimes I am getting a green screen until I hit play. This is random - even with no OFX or TRN.

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 1:52 am
by Dwaine Maggart
Be aware that once you see that message, the GPU's are in a bad state, and will only recover with a Resolve restart. You shouldn't keep trying to do anything.

Is the message appearing on any particular clip? Or only when rendering? Or as soon as you load the project?

What NVIDIA driver are you using?

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 5:08 am
by Scott Stacy
Yep ... I always restart. Usually, Resolve freezes up when I get that warning. I am on NVIDIA 390.65.

It used to occur all the time upon entering any project until Adam provided this code for Advanced Settings in preferences: Local.GPUPinnedHost.Enable = 0.

It occurs during smart renders, 1/4 of the way into a 10-12 minute film (not on any particular clip mind you or with any specific OFX).

Could this be a Resolve GPU memory management issue? For instance, I will start a smart render after working for a bit, the smart render will start and then as things progress I will get the notification. What I find baffling, is that in 14.1 and 14.2, I used to be able to run 2k timelines with multiple instances of TRN and even 20-30 instances of Beauty Box inside power windows plus Resolve OFX Lens Blurs on other clips and not ever get this warning.

Do you have any thoughts about this green screen issue and occasional static screen that shows up on the viewer, video output, and thumbnails? It goes away when I hit play. It's a random occurrence. I'm not seeing it on Alex, Red, or Ursa footage - just Panasonic Varicam LT 4K log footage. Anyway, this appears to be a 14.3 issues. However, I would rule out a hardware issue. Do you see a problem with the placement of my cards?

Thanks,

Dwaine

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 5:48 am
by Dwaine Maggart
Looking at the manual for your mobo, it appears that slots 1 and 3 can be x16 slots.

But looking at the block diagram for your mobo, it shows slots 1 and 2 as x4 slots and all the rest as x8 slots. So that's a bit of a head scratcher.

You say this didn't happen with Resolve 14.1. Was that on this same mobo with this same GPU complement in the same slots?

If so, then this sort of smells like it could be driver related. Current driver is 391.01. You might give that a try. And if that behaves the same, try an older one, like 388.0

If changing drivers has no effect, go into Resolve Preferences, System - Hardware Configuration, set GPU Mode to CUDA, and GPU Selection Mode to Manual. Then deselect one of the Titan X cards, save the config, and restart Resolve. Then repeat the process and use the other Titan X. Is there any difference with one or the other? If so, this might show if one of the GPU's has an issue.

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 12:05 pm
by Chris Timson
I've had this on my Z840 (2x GTX1080)

It was a constant source of frustration and tried restarting the box each time I was about to play out a 45 min programme 1080/25. Every now and again I was still getting GPU mem full.

I eventually found a work around by instead of rendering at maximum speed, I turned it down to play out at maximum of 25fps. This seems to have solved it for now. Not ideal, but it does seem to play out reliably now.

Chris

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 7:14 pm
by Scott Stacy
I checked the manual and found the Work Station block diagram for the x99-E WS and both list the slots as x16. I believe the non-work station mobo is the diagram you may have seen.

I have been on the same mobo and same GPU set up since 12.5. When the problem emerged around 14.3. My GPUs were in slots 3 and 5, as I had a TB3 card in 2. We thought it was the TB3 card causing problems and moved the GPUs up to 1 and 3 thinking it might be a resource issue, but that did not fix the problem. Is the not the best place for them to be from a resource management standpoint?

There seems to be an intermittent and context (Resolve OFX, Beauty Box, and TRN) issue going on. I rendered out a 8 minute Ultra HD timeline last night, but that footage did not have any TNR and minimal resolve lens blur. However, again, in the past, I could pile on OFX without this warning. I worry about doing any long form with this problem occurring.

I will try moving up to 391.01 and going back to 388.0. However, I think I already tried going back to 388.0. I even went back to an earlier system disk image. I will also try your hardware config experiment, too, and report back.

Re: ASUS x99-E WS user question and "GPU Memory Full"

PostPosted: Fri Mar 09, 2018 7:19 pm
by Scott Stacy
Chris Timson wrote:I've had this on my Z840 (2x GTX1080)

It was a constant source of frustration and tried restarting the box each time I was about to play out a 45 min programme 1080/25. Every now and again I was still getting GPU mem full.

I eventually found a work around by instead of rendering at maximum speed, I turned it down to play out at maximum of 25fps. This seems to have solved it for now. Not ideal, but it does seem to play out reliably now.

Chris


I have my render speed typically set at 25-50 - especially, when there's heavy TNR and OFX on a timeline. It's discouraging to hear that your on a z840 and having to do this. I have always thought that z840s are pretty bullet proof. I was thinking about moving to a Z8. With 2 Titans, on my end, I would think it could handle more than 50fps, but I am on a i76950x 10 core system, which is not as robust as a z840.