Best File Location for Performance

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Best File Location for Performance

PostSat Jan 28, 2023 11:29 pm

Hi All,

I am scratching my head as to what would be the best location for the Source Files, Proxies, Clips and target for rendering.

My system has the following storage locations:

1. Drive C (SSD NVME) - Operating System (Win 11 Pro)
2. Drive D (SSD) - 3 x 2TB SSD configured as RAID5 (4TB usable)
3. Drive R (SSD NVME) - 2TB usable

Drive R has much faster read/write speeds than Drive D because it's NVME and usually RAID5 decrease the R/W speeds.

I usually keep my source files on Drive D as this is the largest drive.
I read in a couple of places that it is not recommended to export/render the final work to the same drive as the source files because it will keep reading and writing to the same drive which could basically create a bottleneck and high transaction volume on the drive.

So that being said, I am contemplating between 5 options:

Scenario 1: Keep the source files + proxies + clips on the same drive D. Export to drive D
Scenario 2: Keep the source files + proxies + clips on the same drive D. Export to drive R
Scenario 3: Keep source files on drive D, generate proxies on drive R and export to drive D
Scenario 4: Keep source files on drive D, generate proxies on drive R and export to drive R
Scenario 5: Copy source files to drive R, create proxies on drive R and export to drive D

I would love to get some thoughts on which of the scenarios would be preferred.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Online

4EvrYng

  • Posts: 595
  • Joined: Sat Feb 19, 2022 12:45 am
  • Real Name: Alexander Dali

Re: Best File Location for Performance

PostSun Jan 29, 2023 12:41 am

It is impossible to give exact answer without conducting an experiment because:

1) Yes, RAID 5 is slower on writes than a single drive -BUT- three drives in RAID 5 will be faster on reads than a single one.
2) Speed difference between NVME and SSD can seem huge but actual real world difference can vary based on pattern of usage.
3) Actual speed of NVME write depends on amount of sustained data, keep writing without giving it a break and speed will start sharply dropping off.
4) Windows likes when I/O is split across multiple volumes so, depending on amount of data, usage pattern, etc., engaging "slower" drive in big picture rather than doing everything on single volume can end up faster overall.

So, if I were you I would do test runs to get correct answer for your exact system.
Offline
User avatar

Uli Plank

  • Posts: 21281
  • Joined: Fri Feb 08, 2013 2:48 am
  • Location: Germany and Indonesia

Re: Best File Location for Performance

PostSun Jan 29, 2023 3:09 am

You don’t give us any details of your formats and workflow. H.264/265 is very easy regarding data rates, but demanding on the decoding hardware, while the other extreme would be uncompressed ArriRAW, which needs very high data rates for smooth playback, but is easy to process.
Caching can also be done in lightweight formats for space and speed or in high quality mezzanine formats to be used for final output.
Finally, are you rendering straight into a distribution format or are you generating a high quality master?

All of this goes into such decisions regarding storage.
No, an iGPU is not enough, and you can't use HEVC 10 bit 4:2:2 in the free version.

Studio 18.6.5, MacOS 13.6.5
MacBook M1 Pro, 16 GPU cores, 32 GB RAM and iPhone 15 Pro
Speed Editor, UltraStudio Monitor 3G, iMac 2017, eGPU
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostSun Jan 29, 2023 7:45 am

4EvrYng wrote:...
3) Actual speed of NVME write depends on amount of sustained data, keep writing without giving it a break and speed will start sharply dropping off...


I have massive performance trouble with running Puget benchmark from an (NVME) SSD, since the benchmark creates scores by timing rendering to disk. Many of it's benchmarks stall for tens of seconds after just, say, 30% through (benchmarks are max 2 mins rendering, most less than a minute), because of stalled SSD writes. (the SSD use shows 100%, but data throughput is 0).

I use SSD as a scratch disk, but I now think it's better to use it only to store source media, and use a harddrive as scratch since proxies or cache media probably don't require SSD speeds but incoming TIFF sequences, for example, do.

Sent from my Mi 9T using Tapatalk
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Offline

markdv77

  • Posts: 1
  • Joined: Fri Sep 16, 2022 9:51 am
  • Real Name: Mark de Vries

Re: Best File Location for Performance

PostSun Jan 29, 2023 11:40 am

Speaking as a sysadmin, and not someone with a lot - or any - experience tuning for resolve specifically:
bluemanta wrote:Hi All,

I am scratching my head as to what would be the best location for the Source Files, Proxies, Clips and target for rendering.

My system has the following storage locations:

1. Drive C (SSD NVME) - Operating System (Win 11 Pro)
2. Drive D (SSD) - 3 x 2TB SSD configured as RAID5 (4TB usable)
3. Drive R (SSD NVME) - 2TB usable

Drive R has much faster read/write speeds than Drive D because it's NVME and usually RAID5 decrease the R/W speeds.

I usually keep my source files on Drive D as this is the largest drive.
I read in a couple of places that it is not recommended to export/render the final work to the same drive as the source files because it will keep reading and writing to the same drive which could basically create a bottleneck and high transaction volume on the drive.

Last part makes a lot of sense if whatever you do in between reading and writing is not a significant bottleneck. But if the rate at which the renderer - or proxy generator - consumes and produces data is limited by CPU/GPU, to rates that can easily be handled by the disk, it might not be an issue in practice.
Whether or not it is a bottleneck in your case should be easy enough to test. Put everything on D, render and open the task manager to see Disk and CPU/GPU usage. If the disk goes (close to) 100% percent (and CPU/GPU are not saturated) switch to exporting to R and try again. If disk busy % drops (and CPU/GPU use increases) the disk was a bottleneck.
bluemanta wrote:So that being said, I am contemplating between 5 options:

Scenario 1: Keep the source files + proxies + clips on the same drive D. Export to drive D
Scenario 2: Keep the source files + proxies + clips on the same drive D. Export to drive R
Scenario 3: Keep source files on drive D, generate proxies on drive R and export to drive D
Scenario 4: Keep source files on drive D, generate proxies on drive R and export to drive R
Scenario 5: Copy source files to drive R, create proxies on drive R and export to drive D

I would love to get some thoughts on which of the scenarios would be preferred.


Depending on whether or not the D drive turned out to be a bottleneck I'd go for either 1 or 2. Because I like to keep the originals and proxies together for convenience, and resolve is likely to read from one or the other, not both at the same time, so I'd doubt distributing them over different disk would make a big performance difference.

You did not mention caching... If you are using the render cache it might be worth putting that on one of the faster non-raid disks.

HTH,
Mark.
Offline

Jim Simon

  • Posts: 29806
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Best File Location for Performance

PostSun Jan 29, 2023 3:35 pm

I use the following approach, which offers both good speed and organization.

System
Libraries (Databases)
Export
Cache/Galley/Proxies
Media

The first two can be relatively small - NVMe or SSD are good choices. You may want to consider much larger drives for the latter three, depending on your media formats and frequency of work. I use 10 TB HDD's myself.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostSun Jan 29, 2023 5:17 pm

I like to keep the whole project within a single directory on a single drive - database, source footage and everything else that makes the project, for portability. Fragmenting the project makes everything untidy and in my mind :(

These SSD stalls are quite a drag.
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Online

4EvrYng

  • Posts: 595
  • Joined: Sat Feb 19, 2022 12:45 am
  • Real Name: Alexander Dali

Re: Best File Location for Performance

PostSun Jan 29, 2023 7:39 pm

Mario Kalogjera wrote:
4EvrYng wrote:...
I have massive performance trouble with running Puget benchmark from an (NVME) SSD, since the benchmark creates scores by timing rendering to disk. Many of it's benchmarks stall for tens of seconds after just, say, 30% through (benchmarks are max 2 mins rendering, most less than a minute), because of stalled SSD writes. (the SSD use shows 100%, but data throughput is 0).

I use SSD as a scratch disk, but I now think it's better to use it only to store source media, and use a harddrive as scratch since proxies or cache media probably don't require SSD speeds but incoming TIFF sequences, for example, do.

How much sustained data writes it takes before SSD/NVMe drive starts throttling down and how fast it will recover heavily depends on model of the drive (type of cache it uses) -AND- size of a drive (bigger drive = more cache and better memory layout = it will take longer).

Thus "stalls" you are experiencing are possibly related to your choice of SSD drive so if I were you I would consider changing the drive itself. Look for reviews that measure sustained performance, like ones at https://www.tomshardware.com/reviews/sa ... ssd-review and keep in mind to look for review that compares performance of different capacities of same model.
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 12:43 am

4EvrYng wrote:It is impossible to give exact answer without conducting an experiment because:

1) Yes, RAID 5 is slower on writes than a single drive -BUT- three drives in RAID 5 will be faster on reads than a single one.
2) Speed difference between NVME and SSD can seem huge but actual real world difference can vary based on pattern of usage.
3) Actual speed of NVME write depends on amount of sustained data, keep writing without giving it a break and speed will start sharply dropping off.
4) Windows likes when I/O is split across multiple volumes so, depending on amount of data, usage pattern, etc., engaging "slower" drive in big picture rather than doing everything on single volume can end up faster overall.

So, if I were you I would do test runs to get correct answer for your exact system.


Thank you @4EvrYng, yes I'm experimenting but was hoping to get some ideas from other user's experiences.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 12:49 am

Uli Plank wrote:You don’t give us any details of your formats and workflow. H.264/265 is very easy regarding data rates, but demanding on the decoding hardware, while the other extreme would be uncompressed ArriRAW, which needs very high data rates for smooth playback, but is easy to process.
Caching can also be done in lightweight formats for space and speed or in high quality mezzanine formats to be used for final output.
Finally, are you rendering straight into a distribution format or are you generating a high-quality master?

All of this goes into such decisions regarding storage.


Most of the source material is shot with h.264/265 compression, usually at 4K resolution.

The goal is to produce a hi-res master file and then export to scaled down resolution for different media platforms.

Also, since I'm new to DR, i'm still using the free version which - if I understand correctly - is not taking advantage of hardware decoding/encoding but I'm not sure this is true. In any case, I'm planning to switch to the paid version soon.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:01 am

Jim Simon wrote:I use the following approach, which offers both good speed and organization.

System
Libraries (Databases)
Export
Cache/Galley/Proxies
Media

The first two can be relatively small - NVMe or SSD are good choices. You may want to consider much larger drives for the latter three, depending on your media formats and frequency of work. I use 10 TB HDD's myself.


So, if I understand correctly, you are using 4 separate drives?
Since my Media drive (D) is configured as RAID5, I can add more drives if needed.
My motherboard supports up to 7 SATA 6Gbps channels, so I have some room to grow.
Usually after completing a project, I archive it to an external storage, so my work computer stays somewhat clean for ongoing projects.

So, you recommend a separate drive for each type of files...
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:15 am

Mario Kalogjera wrote:
4EvrYng wrote:...
3) Actual speed of NVME write depends on amount of sustained data, keep writing without giving it a break and speed will start sharply dropping off...


I have massive performance trouble with running Puget benchmark from an (NVME) SSD, since the benchmark creates scores by timing rendering to disk. Many of it's benchmarks stall for tens of seconds after just, say, 30% through (benchmarks are max 2 mins rendering, most less than a minute), because of stalled SSD writes. (the SSD use shows 100%, but data throughput is 0).

I use SSD as a scratch disk, but I now think it's better to use it only to store source media, and use a harddrive as scratch since proxies or cache media probably don't require SSD speeds but incoming TIFF sequences, for example, do.

Sent from my Mi 9T using Tapatalk


Which SSD/NVME drives are you using?
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline
User avatar

Marc Wielage

  • Posts: 10901
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:18 am

Jim Simon wrote:I use the following approach, which offers both good speed and organization.

System
Libraries (Databases)
Export
Cache/Galley/Proxies
Media

We do something similar to what Jim does:

OS & Applications - 2TB SSD boot drive
Resolve Project Server - separate computer & internal SSD
Export (what we call the "Render Drive") - external 8TB spinning drive
Gallery stills / Cache - internal 4TB SSD
Source Media - external 16TB SSD RAID0 [for current projects]

I believe the SSDs are Samsung QV0 870s. We use a RAID0 to push files at at least 2000MB/s, and it works pretty well. Because RAID0's are a little insecure, we back them up on one of two fast RAID5 spinning drives. We generally get pretty good performance from working this way.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:26 am

4EvrYng wrote:
Mario Kalogjera wrote:
4EvrYng wrote:...
I have massive performance trouble with running Puget benchmark from an (NVME) SSD, since the benchmark creates scores by timing rendering to disk. Many of it's benchmarks stall for tens of seconds after just, say, 30% through (benchmarks are max 2 mins rendering, most less than a minute), because of stalled SSD writes. (the SSD use shows 100%, but data throughput is 0).

I use SSD as a scratch disk, but I now think it's better to use it only to store source media, and use a harddrive as scratch since proxies or cache media probably don't require SSD speeds but incoming TIFF sequences, for example, do.

How much sustained data writes it takes before SSD/NVMe drive starts throttling down and how fast it will recover heavily depends on model of the drive (type of cache it uses) -AND- size of a drive (bigger drive = more cache and better memory layout = it will take longer).

Thus "stalls" you are experiencing are possibly related to your choice of SSD drive so if I were you I would consider changing the drive itself. Look for reviews that measure sustained performance, like ones at https://www.tomshardware.com/reviews/sa ... ssd-review and keep in mind to look for review that compares performance of different capacities of same model.


I agree, the type of SSD/NVME plays significant role in the read/write performance.
The Samsung 980 Pro series is great option with rating of 7000 MB/s sequential reads and 5100 MB/s sequential writes. I have seen some SSD from PNY with 7500 MB/s sequential writes.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:31 am

Marc Wielage wrote:
Jim Simon wrote:I use the following approach, which offers both good speed and organization.

System
Libraries (Databases)
Export
Cache/Galley/Proxies
Media

We do something similar to what Jim does:

OS & Applications - 2TB SSD boot drive
Resolve Project Server - separate computer & internal SSD
Export (what we call the "Render Drive") - external 8TB spinning drive
Gallery stills / Cache - internal 4TB SSD
Source Media - external 16TB SSD RAID0 [for current projects]

I believe the SSDs are Samsung QV0 870s. We use a RAID0 to push files at at least 2000MB/s, and it works pretty well. Because RAID0's are a little insecure, we back them up on one of two fast RAID5 spinning drives. We generally get pretty good performance from working this way.


Ha... this sounds like a dream setup :D

By the way, did you mean 2 x RAID5 arrays? because RAID5 requires minimum of 3 drives (1 is used for parity and indexing)
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Online

4EvrYng

  • Posts: 595
  • Joined: Sat Feb 19, 2022 12:45 am
  • Real Name: Alexander Dali

Re: Best File Location for Performance

PostMon Jan 30, 2023 3:05 am

bluemanta wrote:I agree, the type of SSD/NVME plays significant role in the read/write performance.
The Samsung 980 Pro series is great option with rating of 7000 MB/s sequential reads and 5100 MB/s sequential writes. I have seen some SSD from PNY with 7500 MB/s sequential writes.

Some things to keep in mind:

1. Sequential read/write speeds are far from indicator of complete picture. Those are figures you get when synthetic benchmark saturates channel with high queue depth. In real world low queue depths, random access, latencies ... all play important roles and only way to get answer what matters to you is by benchmarking your own setup. Check out Puget Systems benchmark download.

2. Figures you are quoting are theoretical maximums for PCIe 4.0 with 4 free lanes straight to CPU. If your system doesn't have that you won't get those maximums. Also, problem with many "Web benchmarks" is that you are comparing figures without knowing anything about system behind them (how many CPUs and which ones it has). System with 8 cores will result in higher speed score than system with 6. System with 6GHz cores will score higher disk speed than system with 5GHz ones.

3. If your application and system (CPU, memory, etc.) can't saturate drive then its theoretical maximum speed doesn't matter. Can Resolve on your system really write that many GB/sec? It probably can't. Are you reading that many GB/sec that it will make a difference? You probably don't. So get a solid drive but don't obsess over figures too much.

4. Personally I go only for one of big brands (Samsung, WD, Seagate ... with Samsung being my favorite). Life is too short and time is too precious to deal with anything else. I personally don't consider PNY in same category as ones I mentioned.
Offline
User avatar

Uli Plank

  • Posts: 21281
  • Joined: Fri Feb 08, 2013 2:48 am
  • Location: Germany and Indonesia

Re: Best File Location for Performance

PostMon Jan 30, 2023 3:31 am

It's correct that even good SSDs slow down a bit with very long continuous writes.
I have a 2TB Samsung 980 Pro in an Acasis enclosure, connected by Thunderbolt.
It writes a 5 minute section in 23 seconds (28.2 GB). 50 minutes of the same project are written in 261 seconds (285 GB). Nothing to really worry about, but there are differences.
No, an iGPU is not enough, and you can't use HEVC 10 bit 4:2:2 in the free version.

Studio 18.6.5, MacOS 13.6.5
MacBook M1 Pro, 16 GPU cores, 32 GB RAM and iPhone 15 Pro
Speed Editor, UltraStudio Monitor 3G, iMac 2017, eGPU
Offline

Andy Mees

  • Posts: 3208
  • Joined: Wed Aug 22, 2012 7:48 am

Re: Best File Location for Performance

PostMon Jan 30, 2023 11:16 am

Here's my storage setup (just to add to the general data):

1TB NVMe - System + Applications + Databases (Local)
1TB NVMe - Cache/Gallery + Exports
4TB NVMe - Media (Current Projects) <<< everything backed up offline
32TB HDD RAID - Autosave / Backups + Evergreen + Archive <<< everything backed up offline

For me, storage today is already more than fast enough for my work. My storage map is more to avoid single points of catastrophic failure than for gaining specific performance advantages.
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostMon Jan 30, 2023 1:21 pm

bluemanta wrote:
Which SSD/NVME drives are you using?


Kingston A2000, 500 GB, it's in my sig.
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Offline

Jim Simon

  • Posts: 29806
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Best File Location for Performance

PostMon Jan 30, 2023 3:35 pm

bluemanta wrote:So, if I understand correctly, you are using 4 separate drives?
5 actually.

1. System
2. Libraries
3. Exports
4. Cache
5. Media
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 11:00 pm

Mario Kalogjera wrote:
bluemanta wrote:
Which SSD/NVME drives are you using?


Kingston A2000, 500 GB, it's in my sig.


Sorry I missed it in your signature.
I have not tested the WD with DR yet.

I can't test it yet with Puget Benchmark because I'm still on the free version of DR. However, just by comparing the specs and looking at UserBench results, which can still lend some idea about the expected performance.
it looks like this drive, is on the lower spectrum of performance with 1.3/1.1 GBps of read/write. Compared to 2.6/2.8 GBps of read/write on the Samsung 980 Pro series.

Here are links to the benchmarks:

Kingston SA2000M - https://ssd.userbenchmark.com/SpeedTest/882378/NVMe-KINGSTON-SA2000M

Samsung 980 Pro - https://ssd.userbenchmark.com/SpeedTest/1302577/Samsung-SSD-980-PRO-1TB

Obviously Puget benchmark testing on your system will tell a more accurate story but according to Tom's Hardware stress testing on the Samsung 980 Pro series, it held up saturation pretty nicely for 100GB at 4.5 GBps (for 15 minutes write) before performance degraded.
https://www.tomshardware.com/reviews/samsung-980-pro-m-2-nvme-ssd-review

You may want to consider upgrading the Kingston to a better performing NVME drive.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostMon Jan 30, 2023 11:35 pm

4EvrYng wrote:1. Sequential read/write speeds are far from indicator of complete picture. Those are figures you get when synthetic benchmark saturates channel with high queue depth. In real world low queue depths, random access, latencies ... all play important roles and only way to get answer what matters to you is by benchmarking your own setup. Check out Puget Systems benchmark download.


Indeed I'm yet to run the Puget benchmark once I switch to the paid version as it would not run on the free version of DR. Nevertheless, from reading some extensive stress-test results on the Samsung 980 Pro series it looks like it is still hanging at the top of the pack.

4EvrYng wrote:2. Figures you are quoting are theoretical maximums for PCIe 4.0 with 4 free lanes straight to CPU. If your system doesn't have that you won't get those maximums. Also, problem with many "Web benchmarks" is that you are comparing figures without knowing anything about system behind them (how many CPUs and which ones it has). System with 8 cores will result in higher speed score than system with 6. System with 6GHz cores will score higher disk speed than system with 5GHz ones.


Most new motherboards with support for PCIe 4.x and Z690 chipset (Intel) or later support at least 1 or 2 hyper M.2 sockets with 64Gb/s direct lanes to CPU.
Most benchmarks that I read, details the system specifications as well as the load on each test.
Of course, different systems would yield different results, but you can still compare the performance of NVME or SSD drives. http://www.userbench.com collects benchmarks as well as system configuration from all users so you can see the variations of results with different systems. Yet, the differences are not significant, and the results do show a good picture of performance.

4EvrYng wrote:3. If your application and system (CPU, memory, etc.) can't saturate drive then its theoretical maximum speed doesn't matter. Can Resolve on your system really write that many GB/sec? It probably can't. Are you reading that many GB/sec that it will make a difference? You probably don't. So get a solid drive but don't obsess over figures too much.

The capacity to handle saturation, including cache recovery depends on many factors like the size of static cache, "smart cache" on the NVME, the speed of the cache partition, the material and other factors related to the NVME drive itself. There are several benchmark stress-testing tools that are used by different sites so I wouldn't assume the numbers are theoretical only.

4EvrYng wrote:4. Personally I go only for one of big brands (Samsung, WD, Seagate ... with Samsung being my favorite). Life is too short and time is too precious to deal with anything else. I personally don't consider PNY in same category as ones I mentioned.

I couldn't agree more ;)
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostTue Jan 31, 2023 7:30 am

@bluemanta: I'm aware the Kingston is not the best SSD in the world but it's dropping the speed too soon, makes me wander if there is something else that needs adjusting. I must look at the temps, I missed that, perhaps it's overheating. I have first observed the problem when I ran the benchmarks manually, by opening individual projects and rendering. Thought that running it properly from the "main menu" may have a positive impact but it had not.

I see a lot of Puget benchmark results using free version of Resolve so it should work. What are your symptoms?

Sent from my Mi 9T using Tapatalk
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Online

4EvrYng

  • Posts: 595
  • Joined: Sat Feb 19, 2022 12:45 am
  • Real Name: Alexander Dali

Re: Best File Location for Performance

PostTue Jan 31, 2023 9:08 pm

bluemanta wrote:
4EvrYng wrote:1. Sequential read/write speeds are far from indicator of complete picture. Those are figures you get when synthetic benchmark saturates channel with high queue depth. In real world low queue depths, random access, latencies ... all play important roles and only way to get answer what matters to you is by benchmarking your own setup. Check out Puget Systems benchmark download.


Indeed I'm yet to run the Puget benchmark once I switch to the paid version as it would not run on the free version of DR. Nevertheless, from reading some extensive stress-test results on the Samsung 980 Pro series it looks like it is still hanging at the top of the pack.

4EvrYng wrote:2. Figures you are quoting are theoretical maximums for PCIe 4.0 with 4 free lanes straight to CPU. If your system doesn't have that you won't get those maximums. Also, problem with many "Web benchmarks" is that you are comparing figures without knowing anything about system behind them (how many CPUs and which ones it has). System with 8 cores will result in higher speed score than system with 6. System with 6GHz cores will score higher disk speed than system with 5GHz ones.


Most new motherboards with support for PCIe 4.x and Z690 chipset (Intel) or later support at least 1 or 2 hyper M.2 sockets with 64Gb/s direct lanes to CPU.
Most benchmarks that I read, details the system specifications as well as the load on each test.
Of course, different systems would yield different results, but you can still compare the performance of NVME or SSD drives. http://www.userbench.com collects benchmarks as well as system configuration from all users so you can see the variations of results with different systems. Yet, the differences are not significant, and the results do show a good picture of performance.

4EvrYng wrote:3. If your application and system (CPU, memory, etc.) can't saturate drive then its theoretical maximum speed doesn't matter. Can Resolve on your system really write that many GB/sec? It probably can't. Are you reading that many GB/sec that it will make a difference? You probably don't. So get a solid drive but don't obsess over figures too much.

The capacity to handle saturation, including cache recovery depends on many factors like the size of static cache, "smart cache" on the NVME, the speed of the cache partition, the material and other factors related to the NVME drive itself. There are several benchmark stress-testing tools that are used by different sites so I wouldn't assume the numbers are theoretical only.

4EvrYng wrote:4. Personally I go only for one of big brands (Samsung, WD, Seagate ... with Samsung being my favorite). Life is too short and time is too precious to deal with anything else. I personally don't consider PNY in same category as ones I mentioned.

I couldn't agree more ;)


Unfortunately I am under time pressure so I won't be able to address everything in your post in detail but very briefly:

1. When I warn about benchmarks it is to remind everyone reading these posts that variations in figures are introduced through variables that often aren't disclosed, or disclosed in enough detail, that the figures in them can't be treated as absolute gospel. IMHO once "leaders of the pack" are identified any small (percentage wise) difference in figures should be ignored and other attributes should be looked at to make final decision.

2. Theoretical maximum speed of 64GB/sec you quoted is for PCIe 5.0 components using 16 lanes of PCIe 5.0 bus. Current NVMe drives use 4 lanes and are PCIe 4.0 which means theoretical maximum is 8GB/sec. Once overhead is taken into account it is less than that in real world which is why leaders of the pack max out at around 7GB/sec.

3. For this 7GB/sec to matter both application and rest of the components in the computer need to be able of generating 7GB/sec of data writes/reads over long period of time. How many of systems out there (in the hands of mere mortals) running Resolve are capable of doing that? I am guessing next to none. Disk benchmarks can do it but not a real world apps.

Let me put it this way: I am using drive that is slower than yours (Samsung 970 Evo Plus, which is PCIe 3.0 drive on a PCIe 3.0 bus with theoretical max of "only" 4GB/sec) yet not once so far I wished for more drive speed when using Resolve, it is GPU and CPU that are my bottleneck. That is why I am saying that once one identifies which layouts are good (or not) further focus on drive benchmarks and figures becomes more theoretical than practical.
Last edited by 4EvrYng on Wed Feb 01, 2023 12:39 am, edited 1 time in total.
Online

4EvrYng

  • Posts: 595
  • Joined: Sat Feb 19, 2022 12:45 am
  • Real Name: Alexander Dali

Re: Best File Location for Performance

PostTue Jan 31, 2023 9:13 pm

Mario Kalogjera wrote:I'm aware the Kingston is not the best SSD in the world but it's dropping the speed too soon

What makes you believe it is dropping speed too soon? If you do sustained write after how many GB it starts dropping speed and what specs and benchmarks for your model say it should be dropping it at?
Offline
User avatar

Marc Wielage

  • Posts: 10901
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Best File Location for Performance

PostWed Feb 01, 2023 12:56 am

4EvrYng wrote:Let me put it this way: I am using drive that is slower than yours (Samsung 970 Evo Plus, which is PCIe 3.0 drive on a PCIe 3.0 bus with theoretical max of "only" 4GB/sec) yet not once so far I wished for more drive speed when using Resolve, it is GPU and CPU that are my bottleneck. That is why I am saying that once one identifies which layouts are good (or not) further focus on drive benchmarks and figures becomes more theoretical than practical.

How does it measure with the Resolve Speed Test app? Try that and see what it says -- I find it's a good barometer of potential performance.

There are also free Resolve Standard Candle Tests out there: straightforward projects and media that you can run on different machines to get a real-world idea of how they compare.

DaVinci Resolve 'Standard Candle UHD': Rocket Science
https://forums.macrumors.com/threads/da ... e.2273175/
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostWed Feb 01, 2023 4:18 am

Mario Kalogjera wrote:@bluemanta: I'm aware the Kingston is not the best SSD in the world but it's dropping the speed too soon, makes me wander if there is something else that needs adjusting. I must look at the temps, I missed that, perhaps it's overheating. I have first observed the problem when I ran the benchmarks manually, by opening individual projects and rendering. Thought that running it properly from the "main menu" may have a positive impact but it had not.

Thanks for clarifying. Was it working well and then started to drop or did you experience these drops from the beginning?

Mario Kalogjera wrote:I see a lot of Puget benchmark results using free version of Resolve so it should work. What are your symptoms?

Oh that's interesting. I didn't even try. Just saw their statement on the download page but after reading it again, I realized it was referring to the Extended Assets.
Just ran the first Standard test and got overall score of 1530 but it looks like it was only testing on the RAID5 volume.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostWed Feb 01, 2023 11:14 am

bluemanta wrote:
...Thanks for clarifying. Was it working well and then started to drop or did you experience these drops from the beginning?...


I never suspected low SSD performance because there was no indication except perhaps low 4K media results which I attributed to AMD RX580 slow h264 encoding, until I ran benchmarks manually and noticed the playhead stopped moving for a considerable ammount of time...don't know if I can check the file size written at the moment it stops writing i.e. stalling...
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Offline

Andrew Kolakowski

  • Posts: 9209
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Best File Location for Performance

PostWed Feb 01, 2023 12:21 pm

bluemanta wrote:
Uli Plank wrote:You don’t give us any details of your formats and workflow. H.264/265 is very easy regarding data rates, but demanding on the decoding hardware, while the other extreme would be uncompressed ArriRAW, which needs very high data rates for smooth playback, but is easy to process.
Caching can also be done in lightweight formats for space and speed or in high quality mezzanine formats to be used for final output.
Finally, are you rendering straight into a distribution format or are you generating a high-quality master?

All of this goes into such decisions regarding storage.


Most of the source material is shot with h.264/265 compression, usually at 4K resolution.

The goal is to produce a hi-res master file and then export to scaled down resolution for different media platforms.

Also, since I'm new to DR, i'm still using the free version which - if I understand correctly - is not taking advantage of hardware decoding/encoding but I'm not sure this is true. In any case, I'm planning to switch to the paid version soon.


Those are basically never >50MB/sec (typically more like 20-30MB/sec) so even single spinning drive will do, specially if you use separate for read/write.
If you have performance issues then forget about storage as problem lies elsewhere and in this case it's probably decoding/encoding speed (or GPU processing).
You could get into storage speed problems with UHD when using DPX/EXR/TIFF sequences.
Typical h264/5 out of camera have crazy low storage needs. This is not rocket science and trying too hard is just waste of time.
Of course this assumes drives behave properly, but this can be measured even with BM own disks speed test. It may not be 100% reliable test, but gives an indication if all is fine or not.
Offline

SkierEvans

  • Posts: 970
  • Joined: Wed Jan 24, 2018 9:59 pm
  • Location: Ottawa, Ontario
  • Real Name: Ron Evans

Re: Best File Location for Performance

PostWed Feb 01, 2023 1:31 pm

When I had my GH5 and GH5S both shooting UHD 60P at 150Mbps and AX100 shooting HD at 50Mbps I have all of the source files on a 6T hard drive and had no problems running multicam in HD project. My 1080Ti in the PC could manage decode well for this and no speed problems for a simple hard drive. Moving to the GH6 5.7K 60P 10 bit h265 files and things did not work well as then the Threadripper had to do the decode for these h265 files. For Resolve your GPU needs to be able to decode the source files to run well. For h264 and h265 I am sure a cheap hard drive is fast enough. However a cheap/old GPU and CPU is not going to work. I do not export to that hard drive as well though I export to another drive in my case an SSD.

Further, I now put source files on a 2T Western Digital Blue SSD with converter to USB C so that I can use on the Mac and PC. So GH6 4.7 K 60P 10 bit 4:2:0 ( or UHD 10 bit 4:2:2 ) 200Mbps, GH5S UHD 60P 150 Mbps and AX100. These are two hour shows so big files. Runs multicam on Studio Max plugged into USB front connectors no problems. Can take to PC and run project with dropped frames because of GH6 files but does run. GH6 files on PC run with stutters at between 57 to 58 fps so not quite realtime. I do not think drives are your problem.

Also what cameras are the h264 and h265 files coming from and are they 10 bit 4:2:0 or 10bit 4:2:2 as that will make a difference.
Threadripper 1920, Gigabyte X399 DESIGNARE EX, 32G RAM, Gigabyte 4070Ti 12G, ASUS PB328Q, IP4K, WIN10 Pro 22H2, Speed Editor

Resolve Studio 18, EDIUS 9WG,EDIUS X WG, Vegas 18

Studio Max M1 24 core GPU, 32G, 1T drive. iPad Pro 12.9` M2 16G, 1T
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostWed Feb 01, 2023 7:19 pm

4EvrYng wrote:
Mario Kalogjera wrote:I'm aware the Kingston is not the best SSD in the world but it's dropping the speed too soon

What makes you believe it is dropping speed too soon? If you do sustained write after how many GB it starts dropping speed and what specs and benchmarks for your model say it should be dropping it at?


The stall happens anywhere between 5 and 9 GB written....I can see the file size increase until it stalls, then the file size doesn't increase until it's done, when it jumps to the final size. Also happens when copying large files from SATA SSD to the NVME SSD. I don't know when it's supposed to be dropping speed, if ever i.e hopefully not for whole half a minute :D Guess it's just the cache being topped up and slow NAND not being fast enough to offload it?
I have just now achieved the record GPU benchmark with Resolve 18 (praise BMD for performance improvements) but any 4K media benchmark featuring DNxHR is even more pathetic than with R17 before.
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostWed Feb 01, 2023 11:17 pm

Andrew Kolakowski wrote:[

Those are basically never >50MB/sec (typically more like 20-30MB/sec) so even single spinning drive will do, specially if you use separate for read/write.
If you have performance issues then forget about storage as problem lies elsewhere and in this case it's probably decoding/encoding speed (or GPU processing).
You could get into storage speed problems with UHD when using DPX/EXR/TIFF sequences.
Typical h264/5 out of camera have crazy low storage needs. This is not rocket science and trying too hard is just waste of time.
Of course this assumes drives behave properly, but this can be measured even with BM own disks speed test. It may not be 100% reliable test, but gives an indication if all is fine or not.


Thanks. No I don't have any performance issues. Just wanted to make sure I am aware of all the requirements and squeeze the most out of the system as I'm relatively new to DR but been building media computers for 15 years.
So far, ran UserBenchmark and Puget and I'm pretty happy with the results.

Exported 5 minutes video in 4K / QuickTime / h.264 /60fps - took about 11 minutes on the RAID5 volume. I am yet to test the export onto the NVME which I speculate will reduce the export time a little
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostWed Feb 01, 2023 11:33 pm

Mario Kalogjera wrote:
4EvrYng wrote:
Mario Kalogjera wrote:I'm aware the Kingston is not the best SSD in the world but it's dropping the speed too soon

What makes you believe it is dropping speed too soon? If you do sustained write after how many GB it starts dropping speed and what specs and benchmarks for your model say it should be dropping it at?


The stall happens anywhere between 5 and 9 GB written....I can see the file size increase until it stalls, then the file size doesn't increase until it's done, when it jumps to the final size. Also happens when copying large files from SATA SSD to the NVME SSD. I don't know when it's supposed to be dropping speed, if ever i.e hopefully not for whole half a minute :D Guess it's just the cache being topped up and slow NAND not being fast enough to offload it?
I have just now achieved the record GPU benchmark with Resolve 18 (praise BMD for performance improvements) but any 4K media benchmark featuring DNxHR is even more pathetic than with R17 before.


It may be all of the above :D
I mean topped cache + slow writing on the NAND + slow recovery of the cache
you definitely have some bottleneck there.

You could try allocating some RAM exclusively for NVME/SSD cache. It should improve performance quite a bit. Have you looked into this app?: https://www.romexsoftware.com/en-us/primo-cache/

If budget allows, I would be curious to see if popping in the Samsung 970 Pro or 980 Pro would shine the light on the system. ;)
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

Andrew Kolakowski

  • Posts: 9209
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Best File Location for Performance

PostWed Feb 01, 2023 11:35 pm

This is all practically meaningless if you work with h264/5 data out of camera.
Offline

Andrew Kolakowski

  • Posts: 9209
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Best File Location for Performance

PostWed Feb 01, 2023 11:36 pm

bluemanta wrote:
Exported 5 minutes video in 4K / QuickTime / h.264 /60fps - took about 11 minutes on the RAID5 volume. I am yet to test the export onto the NVME which I speculate will reduce the export time a little

I doubt it will make any real difference.
Your encode is 2x slower than RT, so it means your disk needs to be able to deliver eg. 20MB/sec. (probably less).
Cheap pendrive will do it :)
You really trying too hard for no reason :)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 12:03 am

Andrew Kolakowski wrote:
bluemanta wrote:
Exported 5 minutes video in 4K / QuickTime / h.264 /60fps - took about 11 minutes on the RAID5 volume. I am yet to test the export onto the NVME which I speculate will reduce the export time a little

I doubt it will make any real difference.
Your encode is 2x slower than RT, so it means your disk needs to be able to deliver eg. 20MB/sec. (probably less).
Cheap pendrive will do it :)
You really trying too hard for no reason :)


LOL.... well this was the last test and attempt.... saved me about 2.5 minutes... oh well
Now I can sit down with a glass of Pino and admire my creation :D :lol:
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline
User avatar

Marc Wielage

  • Posts: 10901
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Best File Location for Performance

PostThu Feb 02, 2023 12:59 am

bluemanta wrote: Ha... this sounds like a dream setup

It is, but it cost some bucks. We bought used systems locally, gutted the RAID enclosures and dropped in bigger, newer 16TB WD Red drives, so boosting them to 100TB each was "relatively" affordable given the cost of a new RAID from Sandisk.

By the way, did you mean 2 x RAID5 arrays? because RAID5 requires minimum of 3 drives (1 is used for parity and indexing)

We have an older 8-drive 100TB G-Speed Shuttle RAID5, and a newer 8-drive 100TB G-Speed Shuttle RAID5, and a TB3 G-Speed SSD RAID0. And we have 2 or 3 other 4-drive G-Speeds as general purpose drives to use for backups, take home, and share with other systems and stuff.
Image
All told, I think we have at least $30,000' worth of G-Tech drives in the office, so they basically keep us going.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

SkierEvans

  • Posts: 970
  • Joined: Wed Jan 24, 2018 9:59 pm
  • Location: Ottawa, Ontario
  • Real Name: Ron Evans

Re: Best File Location for Performance

PostThu Feb 02, 2023 2:01 am

bluemanta wrote:
Exported 5 minutes video in 4K / QuickTime / h.264 /60fps - took about 11 minutes on the RAID5 volume. I am yet to test the export onto the NVME which I speculate will reduce the export time a little



That is really slow. Using single hard drive source on my PC I can export the GH6 5.7K 60P 10 bits files at just over realtime. At about 57 fps. On the Studio Max going to and from the 2T SSD attached to the USB port it is 1/3 realtime. So I am reading three source files for the multicam edit and writing back a file to the same 2T WD Blue SSD in an adapter to USB port on the Studio Max. I can export to h265 or Prores 422 the whole 1 hour 39 min show in 35 mins. There is something seriously wrong with your system.
Threadripper 1920, Gigabyte X399 DESIGNARE EX, 32G RAM, Gigabyte 4070Ti 12G, ASUS PB328Q, IP4K, WIN10 Pro 22H2, Speed Editor

Resolve Studio 18, EDIUS 9WG,EDIUS X WG, Vegas 18

Studio Max M1 24 core GPU, 32G, 1T drive. iPad Pro 12.9` M2 16G, 1T
Offline

John Waldmann

  • Posts: 114
  • Joined: Fri Aug 12, 2022 3:47 am
  • Real Name: John Waldmann

Re: Best File Location for Performance

PostThu Feb 02, 2023 5:12 am

Gtx3060 is a low end card, so that would be limiting render speeds if you use Resolve studio.

Something to think about when you choose to buy the studio version.
It’s possible it is having some effect on render speeds. Even though the free Resolve “doesn’t rely on the GPU” it is possible some aspects of Resolve free do rely on the GPU to some extent.
Offline
User avatar

Uli Plank

  • Posts: 21281
  • Joined: Fri Feb 08, 2013 2:48 am
  • Location: Germany and Indonesia

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:22 am

That's an internet rumour as false as can be. The free version relies as much on the GPU as the Studio version for identical features. If you expect more speed from Studio, you may be disappointed. The only exception is hardware decoding of some recent formats under Windows, if your hardware supports it at all.

The 3060 is not the latest and greatest, but you should not be limited by it for HD and you can even do some UHD too. Temporal functions, like temporal NR, might run out of VRAM in UHD, though.
No, an iGPU is not enough, and you can't use HEVC 10 bit 4:2:2 in the free version.

Studio 18.6.5, MacOS 13.6.5
MacBook M1 Pro, 16 GPU cores, 32 GB RAM and iPhone 15 Pro
Speed Editor, UltraStudio Monitor 3G, iMac 2017, eGPU
Offline

Mario Kalogjera

  • Posts: 1201
  • Joined: Sat Oct 31, 2015 8:44 pm

Re: Best File Location for Performance

PostThu Feb 02, 2023 1:13 pm

I just did a 4K DCI SCOPE short using similarly priced and performing RX 6700, including the incredibly taxing superscale+ speedwarp so it is possible to work even if you don't have the a 4090 (which probably wouldn't do full real-time as well)...you can't be sure however, that the dreaded "GPU memory full" wouldn't come up...
Asus Prime X370-Pro+R7 3700X@PBO+32 GB G.Skill AEGIS DDR-4@3200MHz
Sapphire RX6700 10GB
Adata A400 120GB System,A2000 500GB Scratch SSDs
Media storage:"Always in motion is it"
BMD Mini Monitor 4K
Windows 11 Pro+Resolve Studio 18+Fusion Studio 18
Offline

SkierEvans

  • Posts: 970
  • Joined: Wed Jan 24, 2018 9:59 pm
  • Location: Ottawa, Ontario
  • Real Name: Ron Evans

Re: Best File Location for Performance

PostThu Feb 02, 2023 1:48 pm

John Waldmann wrote:Gtx3060 is a low end card, so that would be limiting render speeds if you use Resolve studio.

Something to think about when you choose to buy the studio version.
It’s possible it is having some effect on render speeds. Even though the free Resolve “doesn’t rely on the GPU” it is possible some aspects of Resolve free do rely on the GPU to some extent.



My 1080Ti is a much older card than the 3060 with less VRAM , his CPU should also be faster than my old Threaderipper 12 core yet there is a big difference in performance. The Threadripper system only has real problems with the GH6 10 bit files and then the Threadripper can decode at close to realtime. For h264 8bit there is no problem and the Threadripper/ 1080Ti can encode to h264 or h265 at close to realtime or just a bit faster. Not as fast as the Studio Max because of the hardware decoders/encoders. For instance the Studio Max encode to h265 that took 35 mins the Threadripper takes 1 hour 5 mins. Still faster than realtime for that mix of 10bit and 8 bit source files. Still source from the same USB C SSD to an internal hard drive. The difference could be that the source files being used are 10bit 4:2:2 and then Intel QS or the Mac will have the advantage over an NVIDIA or AMD GPU. Maybe try disabling the 3060 and make the Intel QS integrated 770 the GPU for Resolve.

He does not says what his edit involves. If he is using NR on everything and Fusion titles overlayed then that is very different requirements. However if it is a simple edit his system has a problem somewhere and not the drives he is worried about. My data is with using just one WD Blue 2T drive over USB C as the only drive reading from and writing to.
Threadripper 1920, Gigabyte X399 DESIGNARE EX, 32G RAM, Gigabyte 4070Ti 12G, ASUS PB328Q, IP4K, WIN10 Pro 22H2, Speed Editor

Resolve Studio 18, EDIUS 9WG,EDIUS X WG, Vegas 18

Studio Max M1 24 core GPU, 32G, 1T drive. iPad Pro 12.9` M2 16G, 1T
Offline

Jim Simon

  • Posts: 29806
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Best File Location for Performance

PostThu Feb 02, 2023 3:15 pm

John Waldmann wrote:some aspects of Resolve free do rely on the GPU to some extent.
Just for clarity, ALL image processing is done on the GPU. Free or Studio doesn't matter.

Encoding and decoding have advantages in Studio. But the image processing is the same.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:01 pm

Marc Wielage wrote:It is, but it cost some bucks. We bought used systems locally, gutted the RAID enclosures and dropped in bigger, newer 16TB WD Red drives, so boosting them to 100TB each was "relatively" affordable given the cost of a new RAID from Sandisk.

I believe the RED are tagged as surveillance which guarantees reliability on a single drive with long writing periods. I usually prefer the RE (Raid Edition) drives from WD. They are more expensive but are rated for 24/7 continuous spinning which is the case when configured in raid. The other drives are not rated for 24/7 spinning. I had some hit or miss with other drives in RAID5 configuration in the past, so I decided to standardize on RE.

Marc Wielage wrote:We have an older 8-drive 100TB G-Speed Shuttle RAID5, and a newer 8-drive 100TB G-Speed Shuttle RAID5, and a TB3 G-Speed SSD RAID0. And we have 2 or 3 other 4-drive G-Speeds as general purpose drives to use for backups, take home, and share with other systems and stuff.
Image
All told, I think we have at least $30,000' worth of G-Tech drives in the office, so they basically keep us going.

Yeah, the G-Speed are awesome. Just wondering, for the office, wouldn't it make sense to build a storage server with something like 32 or 64 drives? or do you need to constantly mobilize them?
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:09 pm

SkierEvans wrote:
bluemanta wrote:
Exported 5 minutes video in 4K / QuickTime / h.264 /60fps - took about 11 minutes on the RAID5 volume. I am yet to test the export onto the NVME which I speculate will reduce the export time a little



That is really slow. Using single hard drive source on my PC I can export the GH6 5.7K 60P 10 bits files at just over realtime. At about 57 fps. On the Studio Max going to and from the 2T SSD attached to the USB port it is 1/3 realtime. So I am reading three source files for the multicam edit and writing back a file to the same 2T WD Blue SSD in an adapter to USB port on the Studio Max. I can export to h265 or Prores 422 the whole 1 hour 39 min show in 35 mins. There is something seriously wrong with your system.


hence why I'm here ;)
Also keep in mind I'm still evaluating the free version of Resolve which is not using the GPU Acceleration. I am going to purchase the license today and then we can compare apples to apples
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:13 pm

John Waldmann wrote:Gtx3060 is a low end card, so that would be limiting render speeds if you use Resolve studio.

Something to think about when you choose to buy the studio version.
It’s possible it is having some effect on render speeds. Even though the free Resolve “doesn’t rely on the GPU” it is possible some aspects of Resolve free do rely on the GPU to some extent.


Just to clarify, I have RTX3060 not GTX3060

When running the rendering/export I monitored the GPU with the NVIDIA performance panel. I did not see it making any extra effort so I believe the free version is not fully utilizing the GPU Acceleration and it is stated also by BMD
Last edited by bluemanta on Thu Feb 02, 2023 6:47 pm, edited 2 times in total.
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:43 pm

Uli Plank wrote:That's an internet rumour as false as can be. The free version relies as much on the GPU as the Studio version for identical features. If you expect more speed from Studio, you may be disappointed. The only exception is hardware decoding of some recent formats under Windows, if your hardware supports it at all.

The 3060 is not the latest and greatest, but you should not be limited by it for HD and you can even do some UHD too. Temporal functions, like temporal NR, might run out of VRAM in UHD, though.


I am overclocking the 3060 at 1980Mhz and it's pretty stable. I try to be more on the conservative side although I successfully tested it also at 2.1Ghz.

According to this comparison at:https://artgrid.io/insights/davinci-resolve-free-vs-studio/, Studio does utilize GPU accelerators compared to the free version:
"...One of the key advantages of the studio version over the free version is its use of GPU acceleration, including being able to use multiple GPUs. The studio version has GPU accelerated encoding and decoding of the widely used H.264 and H.265 formats that can greatly speed up editing and rendering performance..."

Are you saying it is not true?

All that being said, I may upgrade the GPU to 4080/4080ti... just need to get used to the idea of spending another $400.....
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

bluemanta

  • Posts: 32
  • Joined: Tue Jan 17, 2023 11:31 pm
  • Location: Marietta, GA
  • Real Name: Eyal Kattan

Re: Best File Location for Performance

PostThu Feb 02, 2023 6:56 pm

SkierEvans wrote:My 1080Ti is a much older card than the 3060 with less VRAM , his CPU should also be faster than my old Threaderipper 12 core yet there is a big difference in performance. The Threadripper system only has real problems with the GH6 10 bit files and then the Threadripper can decode at close to realtime. For h264 8bit there is no problem and the Threadripper/ 1080Ti can encode to h264 or h265 at close to realtime or just a bit faster. Not as fast as the Studio Max because of the hardware decoders/encoders. For instance the Studio Max encode to h265 that took 35 mins the Threadripper takes 1 hour 5 mins. Still faster than realtime for that mix of 10bit and 8 bit source files. Still source from the same USB C SSD to an internal hard drive. The difference could be that the source files being used are 10bit 4:2:2 and then Intel QS or the Mac will have the advantage over an NVIDIA or AMD GPU. Maybe try disabling the 3060 and make the Intel QS integrated 770 the GPU for Resolve.


I did try using the integrated UHD770 GPU at the beginning and it was badly stuttering, hence why I started this thread. The RTX3060 changed performance completely.

SkierEvans wrote:He does not says what his edit involves. If he is using NR on everything and Fusion titles overlayed then that is very different requirements. However if it is a simple edit his system has a problem somewhere and not the drives he is worried about. My data is with using just one WD Blue 2T drive over USB C as the only drive reading from and writing to.


I'm new to DR so just basic editing with some dissolves and color corrections. Nothing fancy yet.
You can see my first attempt:
Thanks,

Eyal Kattan


MOBO:ASRock Steel Legend Z690
CPU: Intel i7 12700
GPU: NVIDIA RTX 3060 12GB
RAM: 32GB DDR4 3200
DRV1-OS: Samsung 980 Pro NVMe 1TB
DRV2-CACHE: Samsung 980 Pro NVMe 2TB
DRV3-MAIN: 3x SAMSUNG 870 EVO 2TB (RAID5)
Offline

Andrew Kolakowski

  • Posts: 9209
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Best File Location for Performance

PostThu Feb 02, 2023 7:22 pm

Use longer fades, don't use saturated "anime" colors (specially red). Actually white (or rather very bright grey) looks the best on black background. Cuts, fades etc. link to music tempo.
Use higher bitrate for export as you have your fades badly compressed (maybe it's YT, but I think it's already on the source).
Offline

SkierEvans

  • Posts: 970
  • Joined: Wed Jan 24, 2018 9:59 pm
  • Location: Ottawa, Ontario
  • Real Name: Ron Evans

Re: Best File Location for Performance

PostThu Feb 02, 2023 8:31 pm

Did you record 4K 60P on the GoPro9 ? I think that is h265 which may be the reason for your performance issues. Studio should make a difference.
Threadripper 1920, Gigabyte X399 DESIGNARE EX, 32G RAM, Gigabyte 4070Ti 12G, ASUS PB328Q, IP4K, WIN10 Pro 22H2, Speed Editor

Resolve Studio 18, EDIUS 9WG,EDIUS X WG, Vegas 18

Studio Max M1 24 core GPU, 32G, 1T drive. iPad Pro 12.9` M2 16G, 1T
Next

Return to DaVinci Resolve

Who is online

Users browsing this forum: 4EvrYng, Bing [Bot], Geoff Treseder, govind, muzyphotos, panos_mts, peeceful, twonprod, VMFXBV and 219 guests