Optimized media and how it's used

Hi all,

Just a quick question....

I have a multicam project, all DNxHD. All my media files are on a RAID 0 array. I still cannot get enough throughput on that array to edit in real time, so I've generated optimized media for all my MXF clips. The optimized media is on an M2 drive that's fast as you like.

When I'm in Resolve, doing absolutely nothing except admiring my timeline, I can see in Task Manager that the RAID array is being hammered. When I look in Resource Monitor, I can see that Resolve is actually hammering the original media on the RAID array. I definitely ticked the "use optimized media" option in the playback menu.

So my really quick question is.... why? Why does Resolve need to even look at my original media drive? It doesn't contain any media that it should be using right?

In the screen shot below, you can see drive F: (RAID array), drive C: (system) and drive D: (M2 with optimised media).

Resolve 15.3.1 RAID being hammered.png (307.25 KiB) Viewed 2032 times

A few suggestions that may help.

1) In your Project Master settings, I would set the Optimized Media Format and Render cache format to the lightest possible codec like DNxHR SQ or DNxHR LB, and check Enable background caching. I would also check all auto caching of transition, etc. and you can try also setting the optimize resolution to "Half".

2) Next, Generate Optimized Media for all the clips you want optimize. You can do all of them and that is up to you. Every time you change something in #1, you'll have to delete optimize media and do step #2 again.

3) You already said that you have checked "Use Optimized Media if Available". So you're good here.

4) For Render Cache, set it to User or Smart mode.

Thanks for your reply.

I've done all those things prior to my post above. I also had smart caching set up. I'm just wondering why it's still hammering the RAID array. However, after thirty minutes or so, it stopped. So I think maybe it was indexing the files or something? It's still accessing those original files when I play the timeline, but the hard drive access isn't high. It's just spiking ever five seconds or so.

I think in future I'll dial back on the bitrate of DNxHD to start with. That'll help with throughput on my drives. I guess my RAID array performance just isn't as great as I thought it might be. The system is using the motherboard RAID controller and they're not exactly the best implimentation of RAID I've come across.

Thanks for your help.

You're not playing media in a timeline in the posted image. Do that, then look at Task Manager.

You're not playing media in a timeline in the posted image. Do that, then look at Task Manager.

Correct. My point here is, I don't have to actually do anything in Resolve to hammer the drive that should not be hammered. If I play the timeline, it's just as bad.

Have all your files optimised media been already generated? If you have many long files it can take hours, eg. whole night.

Have all your files optimised media been already generated? If you have many long files it can take hours, eg. whole night.

Yes, I think so. I followed the recommendation in the reference manual to set up a smart bin to show me all the files that were optimised. There were none that showed as not optimised.

And it's still doing it?
If dos then this is some problem between Resolve and RAID. Update drivers for RAID.
So activity is 100% and moment you close Resolve it goes down?

And it's still doing it?

Yes. It did it for a good thirty minutes and then stopped. And on other occasions, if it's doing it, then it stops as soon as I exit Resolve.

Note that I'm running Resolve 15.3.1 and all my drivers are up to date.

I believe that it's indexing files and that's why it goes away after thirty minutes or so.

Audio waveforms do get generated from the original media, even when Optimized Media is attached.

Once the activity stops, it doesn't return?

Audio waveforms do get generated from the original media, even when Optimized Media is attached.

Ah right. That is a good answer! The optimised media is coming from the M2 drive, but the original MXF files will still get accessed to get the waveforms or maybe audio. That works for me.

Good thinking!

Once the activity stops, it doesn't return?

And yes... if the drive is being hammered without me doing anything, then it eventually stops and doesn't return. However, the RAID array DOES get access when I play a timeline again, but not heavily.

Okay, this problem came back when I upgraded to Resolve 16.2.2.011. I believe this is the latest version.

To better describe the issue, I've taken a screen capture video, as below.

The system specs are:

Intel i7-5820 @ 3.3GHz
16 Gb DDR4 RAM
Nvidia GTX 780 Ti
500 Gb SSD system drive (C)
500 Gb M2 cache drive (D)
3 x 3 Gb SATA3 media drives in RAID 0 (F)

I also installed a BMD MiniMonitor PCI card which drives a second monitor, but for this thread, the MiniMonitor is not configured in Resolve. I only tend to use it when grading.

Again, all the original media is on the media drive (F). All the optimised media is on the cache drive (D). It looks to me like Resolve is heavily utilising the original media, not the optimised media.

Anyone any ideas? Thanks in advance.

Interesting.... I was doing some more testing with my project, Task Manager and Resource Monitor. The devil is in the detail. It seems there are two issues:

1. Resolve isn't using the optimised media.

I have since deleted it and will generate it again, just in case it lost it. But then I noticed:

2. Resource Monitor clearly shows that when Resolve is playing my timeline, the original media drive is being hammered. However, it appears that it's the SYSTEM process that is doing said hammering. That's weird right?

So, my next test is, I'm switching off "allow files on this drive to have contents indexed in addition to file properties" on my RAID / original media drive. My thinking is that Windows 10 is trying to get its head around those big MXF files on the RAID array, whenever Resolve is trying to use those files or maybe is pulling the audio from those files?

I'm not sure if Resolve uses the original media audio or audio it creates in the optimised media files, but that could perhaps explain the SYSTEM overhead on the original media files?

Seriously, nobody has any idea what could be causing this issue?

Okay, I have some more info for you. I finally nailed it to something small:

1. I load the project.

The project has brand new optimised media and I even defragmented the RAID array, despite the fact it didn't need it. The timeline plays SLOW. Far too slow. It's a multicam edit timeline. Not sure if I mentioned this before? Four cameras. Resolve just cannot keep up, but this is clearly because the RAID array is being hammered (mostly by the SYSTEM process, it has to be said). That is to say, the original media on the RAID array is being hammered but the optimised media on the cache drive (D) is not being touched. Task Manager shows zero disk access, while playing the timeline, on the cache drive.

2. Untick "Use Optimised Media if Available".
3. Tick "Use Optimised Media if Available".
4. Play the multicam timeline.

It now plays fine and I can see that the disk access on the cache drive (where the optimised media lives) is bubbling away.

So... It's a bug in Resolve 16.2.2.011? Have I finally solved it?