14.3 - GPU Memory Full

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

14.3 - GPU Memory Full

PostTue May 29, 2018 9:29 am

Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline

Michael Wade

  • Posts: 11
  • Joined: Tue Feb 20, 2018 3:08 pm
  • Real Name: Michael Wade

Re: 14.3 - GPU Memory Full

PostTue May 29, 2018 2:56 pm

Dany I tried out Resolve 15 and after 20 mins happily editing I got the GPU Full message and everything went to pot. Spec: i7 CPU 860 2.80 GHz (8 CPUs) 16 RAM Card AMD HD Radian 7800 I was thinking of ungrading the graphics card but seeing that your spec is getting the same GPU Full message I wonder if it's worth bothering. Cheers.
GH4 Windows 10 DaVinci Resolve Studio 18.1.1
i7 CPU 2.80 GHz 16 RAM GTX 1060
Offline

Rohit Gupta

Blackmagic Design

  • Posts: 1689
  • Joined: Wed Aug 22, 2012 5:00 am

Re: 14.3 - GPU Memory Full

PostTue May 29, 2018 3:38 pm

Dany Evans wrote:Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany


Logs please?
Rohit Gupta

DaVinci Resolve Software Development
Blackmagic Design
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostWed May 30, 2018 7:59 am

Rohit Gupta wrote:
Dany Evans wrote:Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany


Logs please?


Here link for the logs: http://www.schelby.tv/forum/DaVinci-Resolve-logs-20180530-095008.zip

Thanks again!
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostThu May 31, 2018 11:15 am

And again "GPU MEMORY FULL" - this time on rendering (DNxHD 1080i/mxf).

Please help, thanks!

Log here: http://www.schelby.tv/forum/DaVinci-Resolve-logs-20180531-131254.zip
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline
User avatar

Dwaine Maggart

Blackmagic Design

  • Posts: 13258
  • Joined: Wed Aug 22, 2012 2:53 pm

Re: 14.3 - GPU Memory Full

PostThu May 31, 2018 6:21 pm

First, I'm impressed you are able to make 4 Titans work in this environment.

Is there some reason you are only using 3 of them for GPU processing? Since all the cards match, you should be able to use the GUI card for processing as well.

I only see the GPU mem error in one log, from today, using the ARTE project. It appears that GPU 3 is not happy about something and is not responding properly to CUDA commands.

Since it seems to be only 1 specific card having the issue, there might be an issue with the card itself. Looks like the error may have happened during or just after some rendering, so perhaps an overheating issue?

As a test, go into Resolve Preferences, System - Hardware Configuration, set the GPU processing mode to CUDA, set the GPU selection mode to Manual, and uncheck the 3rd Titan card. Save the prefs and restart Resolve. See if that makes the issue go away.

If not, please provide a new log after the issue repeats.
Dwaine Maggart
Blackmagic Design DaVinci Support
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostFri Jun 01, 2018 7:52 am

Hi Dwain,

Thanks very much!

With internal GPU reslove is much slower as the 3 external cards only. We do the candel test with/without internal Titan. Without it's much more perfotmance, no idea why. Maybe because it's a flashed Mac card, so we used this one just for GUI.

Anyway, we do Geekbench tests and all cards are at perfect score (except the internal one). But as you describe, maybe it's really a overheating issue - we check that out very soon.

Thanks again for your fast help :)
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostTue Jun 05, 2018 2:26 pm

We do some testing and we can say, It's not a demage GPU issue. We change the whole GPU's (from other workstation), but the problem is still exist - also with two TITAN's only. Curious: the message pop up once, freeze the image and after restart the computer Resolve runs without any problems.
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostMon Jul 02, 2018 8:40 am

Stil random "GPU MEMORY FULL" Issue with absolut simple tasks. Restart the machine and the issue is gone - for a while. We kick out 2 of our GPU's and work with 2 only now. Also change the GPU's together, but now way.

I think the best is roll back to 14.02 and hope it solve the problem because.
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2
Offline
User avatar

Uli Plank

  • Posts: 25472
  • Joined: Fri Feb 08, 2013 2:48 am
  • Location: Germany and Indonesia

Re: 14.3 - GPU Memory Full

PostMon Jul 02, 2018 12:59 pm

What about power supply? Can you try to monitor voltage when this happens?
My disaster protection: export a .drp file to a physically separated storage regularly.
www.digitalproduction.com

Studio 19.1.3
MacOS 13.7.4, 2017 iMac, 32 GB, Radeon Pro 580 + eGPU
MacBook M1 Pro, 16 GPU cores, 32 GB RAM, MacOS 14.7.2
SE, USM G3
Offline

Dany Evans

  • Posts: 191
  • Joined: Tue Dec 17, 2013 9:32 am
  • Location: Germany / Swiss

Re: 14.3 - GPU Memory Full

PostTue Jul 03, 2018 6:40 am

Hi Uli,

I don't think it's a power problem. 3 of the Titans are in a expansion box with 2400W. We have stress tested all of them. But for sure we removed 2 of them now. If the issue still there we do a rollback to 14.2.
MP7.1 16 Core 3,2Ghz | OS 13.5 | 192GB Ram | Vega II Duo | Decklink 4K ex 12G with Quad SDI board | DR 18.6.3 B19 | Desktop Video 12.1 | SSD Raid | Advanced Panel 2

Return to DaVinci Resolve

Who is online

Users browsing this forum: Baidu [Spider], FoamDomeProductions, Geoff Treseder, Mads Johansen, panos_mts and 309 guests