Blackmagic Forum

It is really a shame to see the GPU-related error cascade going on and on and on…

Just to get possible doubts out of the way: I joined the Davinci platform with version 14 (always the studio variant and not just one instead 6 or more concurrently), and I was coming from the Adobe side, like many others here…

One of the main reasons for trying to change, was the extremely promising use of multi-GPU hardware, which Adobe hasn`t even heard of at that time. Like many Adobe users, it was anything else than easy, but over time I have got the feeling that my decision was right.

So, with the next 2 Davinci releases 15 & 16, everything went – at least more or less - well but in the later V 16 days, the “GPU failed to perform…” issue was getting more and more prominent. To make the long story short(er) the current situation has changed dramatically, compared to – let`s say – one year ago.

17.2.1 is for sure the most buggy, unstable, and unreliable application, which can`t even be used as a toy, not to speak of professional use (at least on more complex hardware. Not only were we tortured with errors like 209, 700, 702, GPU memory full and others, but today – believe it or not – with Nvidia Studio drivers 471.11 the error 719 showed up, were before 700 and 702 was dominating.

For me, as somebody who uses more advanced systems – one of my current PCs is a 64 Core, 128 GB Ram, 2x RTX 2080 TI, PCIe 4 and SSD` transfer rates of beyond 6000MB/s – my patience is rapidly approaching its end.

I simply can`t see the point any longer, having systems worth of several 10K USD each, but getting forced to restrict the use of each machine to 1 single GPU (preferences config fully manual) while hoping Davinci survives at least half an hour before it gets unusable again.

The truth is, that since autumn last year, no serious work is possible any longer. I have tried and installed (several times) any whatsoever Nvidia driver release since version 456.71 (always tried both, gaming and studio driver, if available) but Davinci remains buggy like hell, which is not just disappointing, but slowly makes me angry.

Blackmagic – is this your understanding of how to treat your paying customers, by completely ignoring the major problem of your software??? Pointing towards others like Nvidia & AMD is your only idea of how to “solve” the longest-lasting construction site within Davinci?

Common Blackmagic, wake up and get these issues under control, instead of flood the place with new features nobody can use, as long as there is no reliable build!!!

If you have your system set up and woking correctly, with an appropriate GPU driver, and don't change anything... it keeps working.. unfortunately windows updates often break this.

for now.. try un installing the GPU driver, then doing a clean full and custom install of the current driver for your GPU.. if that doesn't help, see the FAQs to capture resolve logs and post here with your system info for this logs.

Steps that repo the errors will help someone follow up too.

Dear Peter!

Many thanks for your immediate reply here, it feels good when someone cares about an issue, especially because of the fact, that an even bigger nightmare than having problems with stability, would be to be forced to get back on the Adobe platform.

I am a big fan of systematic approaches when it comes to tackling a specific situation like this, therefore please advise - I have different possibilities in terms of the configuration of the system, so what would you recommend starting with:

1) Studio or Game Ready driver?
2) do you think it is better to go with or without an NV-Link based setup regarding the GPUs – as far as I know by now, in both cases the error(s) appears almost similarly?
3) When it comes to the Davinci internal config: all auto, or manually with Cuda and manually one or two cards selected?
4) And lastly: is it enough to provide the Davinci crash log and the DirectX diagnosis or should I do something in addition to this, besides the detailed hardware description?

The description for the steps to reproduce the error is in all cases very simple:

a) Just moving the playhead in the edit window and if this works without errors then
b) exporting and rendering the timeline triggers the error

Again, many thanks for your help!

Blockcatcher wrote:with Nvidia Studio drivers 471.11 the error 719 showed up, were before 700 and 702 was dominating.

Another post here on the forums mentioned nVidia recommends to install previous driver instead of 47x. Of course if you get another bugs with earlier versions it doesn't really matter jut that the bug is a different one.

Thank you for your contribution:

nother post here on the forums mentioned nVidia recommends to install previous driver instead of 47x. Of course if you get another bugs with earlier versions it doesn't really matter jut that the bug is a different one.

But as I have already mentioned, I have installed absolutely every driver without any exception since 456.71 and always the Game Ready & Studio variant(as long as available). Therefore it is no question whether using the current or the previous driver, it applies to all of them since Davinci Version 17.

Additionally, I am pretty sure, that the errors 700, 702 & 719 are more or less pointing to a very similar if not to the identical cause...

With a lot of these drivers running just one of the two cards works (more or less) at least with much fewer hiccups (700 & 702) but none of them allows 2 cards concurrently to be applied.

1. Studio
2. Without
3. Manual

Can I confirm you installed the old GPU driver, then did
a clean full and custom install
of the current driver for your GPU?

To be historically correct, I tried almost any available driver from 456.71 up to the current one (471.11). Installation was always done as a “new” install, which means official Nvidia installer, first de-installation of the old one, then installed the new one.

Also already mentioned, I always tried studio (when available) as well as the game-ready variant.
“My GPU” is actually 2x RTX 2080Ti, in some cases when selecting just one of them (inside the Davinci preferences), it works for some time, but never longer than a few minutes up to two or three hours before one of the usual suspects (700 or 702) and never with both cards (both cards selected, it doesn`t work at all).

All other apps on the PC are working flawlessly including all the widely known CPU & GPU performance tests. 13K with 3DMark time spy extreme and 27K with Cinebench R20. Due to the 2-loop water cooling system, even demanding apps can`t challenge the (GPU & CPU) hardware at all, it runs perfectly fine and my GPUs never have seen temperatures above 55° Celsius even under extreme load.

ok, grab resolve generated diagnostic logs and post a link here so someone can take a look

Martin

Just throwing out two possible setup issues for multi-GPU systems...

a. Can you check if SLI is enabled? If yes, can you try without SLI?
b. Can you confirm that all your displays are connected to the same GPU?

Please see Dwaine's post here on how to generate and share a link to logs and NFO.

Regards
Shrinivas

Dear Shrinivas!

1) Yes, SLI/NVlink is disabled
2) Yes, both displays are connected to the card in the very first PCIE 16x Slot (Davinci shows it as "Main Display)

I will check Dwaine`s post immediately and undertake the according steps ASAP, many thanks, once done, I`ll be back here with logs and info!

Ok, here you go:

https://we.tl/t-Xc1RoTW48H

in case the URL doesn`t show up here it is with a blanc after the https slashes:

https:// we.tl/t-Xc1RoTW48H

these are 3 files:

- Resolve Logs
- Windows System Information
- DxDiag

Nvidia Studio 471.11, no SLI/Nvlink, both cards to the 1st GPU connected, in DaVinci config manually Cuda & manually both cards selected - looks like the good old error 700 is back...

Blockcatcher wrote:All other apps on the PC are working flawlessly including all the widely known CPU & GPU performance tests. 13K with 3DMark time spy extreme and 27K with Cinebench R20. Due to the 2-loop water cooling system, even demanding apps can`t challenge the (GPU & CPU) hardware at all, it runs perfectly fine and my GPUs never have seen temperatures above 55° Celsius even under extreme load.

nVidia hasn't supported SLI since the beginning of this year.

Beginning on January 1, 2021, no new implicit SLI profiles will be provided for any RTX 2XXX or earlier GPU. All GPU support for SLI going forward will be via explicit SLI, which is to say, the game will have to support the mode directly rather than implementing the mode in-driver.

I'm not sure if existing support ended also, or is it just that no new support will be provided. Or if Resolve has, had or needed specific support before. Anyway, SLI has always been kind of hit&miss, although when worked, it worked well and gave performance boost.

As far as I know, Resolve does not and has not used SLI. SLI is for split framebuffer rendering in OpenGL and DirectX where frame is rendered as parts on multiple gpus and combined on one for display output, but Resolve doesn’t use this kind of processing. NVLink is another story, but last I asked, support said its usage ends with just will not crash ”feature”

Please send a project export .drp of the "Mattenskilauf - Puchberg Losenheim - Wunderwiese - Servus TV - Full Effects 2 AllPlugIns 2" project.

You can PM a link if you don't want it publicly shared.

Dear Dwaine!

At first please let me thank you very much for your help, I am really extremely thankful for this valued, qualified support!

I have sent you a pm with the export project file.

Possibly, there is some light at the end of the tunnel, because by chance I have entered a stage, where I can intentionally either trigger or avoid the error(s).

With exact this particular hard- & software configuration you have the log files of, I today was able to work completely error-free for more than 5 hours, by having (manually) just the “Main Display” 2080TI selected in the DaVinci preferences. This seems to be stable as I was “allowed” to quit and reopen Davinci and even after a computer restart, DaVinci opens and runs stable…

But as soon as I check the box for the second card, an error 700 shows up.

I hope this helps a bit to get closer to a possible solution, again many thanks, I am looking forward to your reply!

Error 700 is cudaErrorIllegalAddress

Loading your project on a dual RTX6000 system, I have no such issues.

But what I do have is the display connected card using 17GB of VRAM. And the other GPU using 15.2 GB of VRAM, per the Windows Task Manager Performance - GPU tab. That's fast forwarding through the project. Likely some clips are using more VRAM than others.

Both are substantially more than the 11GB of VRAM that your RTX2080Ti cards have.

And this is with no media. I would expect it to be higher with actual media being processed.

Why it's worse with 2 cards, I'm not sure. With the project you sent, I would expect 1 GPU to also have GPU mem errors.

At any rate, it appears you are trying to do more than your available VRAM supports.

As a test, make a copy of the project, and set the timline resolution to HD. Does it work then?

Dear Dwaine!

Thank you very much for taking the time to take a closer look, but:

1) in 98% of all cases I get “700” or “702” and not “GPU memory full” errors

2) the provided logs also clearly show (in the provided example) a 700 error and therefore are pointing in this direction.

3) I don`t know what causes the load on your RTX 6000`s, but the attached screenshot – which was taken during playback - with all the media, the PlugIns, color grade, and all the other bells and whistles activated, shows that the workload on the only selected GPU (Main Display), is clearly below 8GB and therefore not problematic at all.

4) Also, to be seen is that the playback runs on full project FPS (23.976), with totally nothing pre-rendered or cached, which is not challenging at all for the card.

5) As far as (at least) I have seen in the logs, nothing justifies the assumption, that the 700 and 702 errors are caused by a GPU memory overload…

You said you don`t know what causes the Cuda addressing related error, but I think exactly this is the main question here:

“Why isn`t possible to use two cards concurrently”, especially when one card solo runs (since yesterday) totally – and much more important – reproducible smooth?

Please don`t get me wrong, I really don`t want to underrate either you’re for sure vast experience or qualification, but due to the fact, that even early DaVinci 17 versions (betas – unfortunately I don`t remember which ones exactly) took advantage of both cards, leads to the assumption, that there must be something relatively new in the code, which causes this issue.

Please help me figure out how to overcome this problem, thank you very much

As I mentioned, dual RTX6000 cards are working fine for me. I didn't mention, but I'm using the current 471.11 driver.

My mem usage test was with fast forwarding through the timeline at x64. If you do that, does your GPU mem usage increase in Task Manager?

To be honest I never used this feature ever before (and probably I won`t need it in the future either) but you said check, so I did. As you can see on the attached screenshot, it puts no additional load on the card – and before you ask: yes, I am using exactly the same driver like you – in the studio variant of course.

Unfortunately, this doesn`t bring us nearer to the solution to my issue.

Don`t you have access to one of your lead software developers, who are capable of interpreting the sent logs in a higher detail, in order to find out why the hell, DaVinci is unable and to integrate and take advantage of a second GPU???

I mean it can`t be satisfying for Blackmagic to advertise with the Multi-GPU feature, which doesn`t work properly in the end (at least in a lot of cases)!

In addition, to clarify also this topic, this particular machine is a pure workhorse, no gaming, no playing around (which especially affects the installed software, just serious professional video & audio apps installed, no kid stuff).

I would even be willing to define a special Windows start-up environment, regarding the active Windows services, just to get the multi GPU up and running, if this helps.

Please be so kind and try to find somebody within Blackmagic, who really has an own interest in bug-fixing this case together with me. I am CTO of a software development company in the field of secure communication (Qnective.com) and therefore I know, developers like to be challenged, especially when the case is somehow extraordinary…

I am sure these Cuda- and closely related errors are a prominent topic on your development road map and here is the chance to find out a bit more…

R both GPUs in x16 slots?
Can u disconnect the two UI monitors and plug one into the other GPU and report how that changes status?

Yes, both GPUs are in a 16Bit PCI-E 4 Slot, and deploying the monitors (one to each card - if I have understood your request correctly) doesn`t change anything. And yes, I have tried this vice versa as well:

monitor A - to card A(PCI-E Slot 1)
monitor B - to card B(PCI-E Slot 3)

as well as

monitor A - to card B(PCI-E Slot 3)
monitor B - to card A(PCI-E Slot 1)

I also have already tried different monitor to card connections, such as:

- Thunderbolt 3 to Thunderbolt 3
- HDMI to HDMI
- display port to display port


Only as long as just one card is selected in the Davinci preferences it works, selecting both, a 700 or 702 error appears (i have no clue why with the exact same setup, sometimes a 700 and sometimes a 702 appears.

Just to let you know, the motherboard is a Gigabyte TRX40 Designare Thunderbolt 3 board, with (of course) the most recent bios installed, the monitors both are (identical) 32" 4K BenQ PD3220U .

only one GPU can be used for monitoring.
so i was after a single monitor plugged into the other GPU, that showed no use previously, and no second monitor.
if my understanding is correct, the original working GPU would not now work, the one that didnt now would?

Dear Peter, Dear Dwaine!

Thank you again for taking care of the issue, which I really appreciate very much. But I think guessing doesn`t take us any further from here.

Is one of you directly involved in the software development process, as a product manager or even better, as a software developer? If not please let me speak to one of these guys. I also offer any kind of remote access to my machine, with TeamViewer or any other remote app you might prefer.

Nevertheless, to try to answer your most recent question, I have done the following:

1) just connected 1 monitor to the “secondary” GPU (the one the 16Bit PCI-E 4 -Slot Number 2).
2) then, in the DaVinci preferences, consequently this is the only GPU that can be selected.
3) just look at the Nvidia control panel shot, which confirms this.

Now running DaVinci works (on this particular card and this particular monitor) exactly as it did when the other possibility was chosen (Main GPU in the 1st 16Bit PCI-E 4 Slot, only one monitor connected to it).

Now please look at the Task Manager!!!
Although just the secondary card is selected in the DaVinci preferences, just the primary GPU (the one in PCI-E 4 - 16Bit Slot 1) gets loaded(addressed) and obviously performs all the work – now tell me please – HOW CAN THIS BE???

Additionally, in these two setup variants, it doesn’t matter at all if a second monitor is connected to the “opposite” card, as long as just 1 GPU is selected in the DaVinci preferences – it simply works stable, although not as powerful as it would with two GPUs.

The problem is still unchanged and always present (even with DaVinci 12.2.2, which I also have tried already) – no chance to take advantage of two concurrently applied GPUs…