Page 1 of 1

14.3 - GPU Memory Full

PostPosted: Tue May 29, 2018 9:29 am
by Dany Evans
Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany

Re: 14.3 - GPU Memory Full

PostPosted: Tue May 29, 2018 2:56 pm
by Michael Wade
Dany I tried out Resolve 15 and after 20 mins happily editing I got the GPU Full message and everything went to pot. Spec: i7 CPU 860 2.80 GHz (8 CPUs) 16 RAM Card AMD HD Radian 7800 I was thinking of ungrading the graphics card but seeing that your spec is getting the same GPU Full message I wonder if it's worth bothering. Cheers.

Re: 14.3 - GPU Memory Full

PostPosted: Tue May 29, 2018 3:38 pm
by Rohit Gupta
Dany Evans wrote:Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany


Logs please?

Re: 14.3 - GPU Memory Full

PostPosted: Wed May 30, 2018 7:59 am
by Dany Evans
Rohit Gupta wrote:
Dany Evans wrote:Hi all,

We have a random GPU MEMORY FULL error. That's new for us, we never have this bevore.

Spec:

- MAC 5.1 12Core (96GB RAM) with 4 x GTX TITAN.
- DR 14.3
- OSX 10.12.6 with latest security update
- Latest Nvidia driver (CUDA 387.178 / GPU DRIVER 378.05.25f08)

I think it's since the last security update. Someone the same issue?

Thanks,
Dany


Logs please?


Here link for the logs: http://www.schelby.tv/forum/DaVinci-Resolve-logs-20180530-095008.zip

Thanks again!

Re: 14.3 - GPU Memory Full

PostPosted: Thu May 31, 2018 11:15 am
by Dany Evans
And again "GPU MEMORY FULL" - this time on rendering (DNxHD 1080i/mxf).

Please help, thanks!

Log here: http://www.schelby.tv/forum/DaVinci-Resolve-logs-20180531-131254.zip

Re: 14.3 - GPU Memory Full

PostPosted: Thu May 31, 2018 6:21 pm
by Dwaine Maggart
First, I'm impressed you are able to make 4 Titans work in this environment.

Is there some reason you are only using 3 of them for GPU processing? Since all the cards match, you should be able to use the GUI card for processing as well.

I only see the GPU mem error in one log, from today, using the ARTE project. It appears that GPU 3 is not happy about something and is not responding properly to CUDA commands.

Since it seems to be only 1 specific card having the issue, there might be an issue with the card itself. Looks like the error may have happened during or just after some rendering, so perhaps an overheating issue?

As a test, go into Resolve Preferences, System - Hardware Configuration, set the GPU processing mode to CUDA, set the GPU selection mode to Manual, and uncheck the 3rd Titan card. Save the prefs and restart Resolve. See if that makes the issue go away.

If not, please provide a new log after the issue repeats.

Re: 14.3 - GPU Memory Full

PostPosted: Fri Jun 01, 2018 7:52 am
by Dany Evans
Hi Dwain,

Thanks very much!

With internal GPU reslove is much slower as the 3 external cards only. We do the candel test with/without internal Titan. Without it's much more perfotmance, no idea why. Maybe because it's a flashed Mac card, so we used this one just for GUI.

Anyway, we do Geekbench tests and all cards are at perfect score (except the internal one). But as you describe, maybe it's really a overheating issue - we check that out very soon.

Thanks again for your fast help :)

Re: 14.3 - GPU Memory Full

PostPosted: Tue Jun 05, 2018 2:26 pm
by Dany Evans
We do some testing and we can say, It's not a demage GPU issue. We change the whole GPU's (from other workstation), but the problem is still exist - also with two TITAN's only. Curious: the message pop up once, freeze the image and after restart the computer Resolve runs without any problems.

Re: 14.3 - GPU Memory Full

PostPosted: Mon Jul 02, 2018 8:40 am
by Dany Evans
Stil random "GPU MEMORY FULL" Issue with absolut simple tasks. Restart the machine and the issue is gone - for a while. We kick out 2 of our GPU's and work with 2 only now. Also change the GPU's together, but now way.

I think the best is roll back to 14.02 and hope it solve the problem because.

Re: 14.3 - GPU Memory Full

PostPosted: Mon Jul 02, 2018 12:59 pm
by Uli Plank
What about power supply? Can you try to monitor voltage when this happens?

Re: 14.3 - GPU Memory Full

PostPosted: Tue Jul 03, 2018 6:40 am
by Dany Evans
Hi Uli,

I don't think it's a power problem. 3 of the Titans are in a expansion box with 2400W. We have stress tested all of them. But for sure we removed 2 of them now. If the issue still there we do a rollback to 14.2.