Blackmagic Forum

I'm a new user and have seen recommendations to set up the database on an external SSD. I want to ask the more experienced users how they configured external storage in their studios???

Whàt did you use that you like and what to avoid.

Sent from my Pixel XL using Tapatalk

I recommend avoiding all externals and using only internal drives.

I favor a five drive setup as a starting point.

System (nothing project related)
Databases
Gallery, Cache, Optimized Media & Proxies
Exports
Media

Are you thinking of an SSD to get better read performance, compared to a single hard drive? Referencing "read" because much of Resolve's performance comes from being able to accessing data faster. I needed faster read speeds (I film in RAW) and have started using a RAID 0 (striping) configuration using larger hard drives and a OWC ThunderBay enclosure. With three HDDs I got about 2.5X read throughput improvement (around 250MB/s to 600MB/s plus). This, combined with proxies made editing much smoother, especially when more than one clip is active.

Some PROs and CONs. A single fast SDD (M.2 NVMe) is cheaper and faster than a RAID 0 config, by a long shot, but stores far less data. Both SSD and RAID 0 configs loose all data if there's a failure. But with a RAID 0 config you might have lots more data to loose! The morale is to have backups--manual copies (for SSD or RAID 0), RAID 1 (mirror) or RAID 5 (can recover from the failure of a single drive). (And the very nervous soul--ahem, me--I'd create a manual backup of a RAID 5 config or use RAID 10/1+0.) For "manual copy" or RAID 1/10 you're looking at 2X cost--a duplicate set of drives. For RAID 5 you're looking at incremental cost--one additional drive.

In all cases SSDs, HDD RAIDs, even SSD RAIDs, remember that your external interface speeds (Thunderbolt x, PCIe) and CPU also contribute to read performance. And external enclosures--for example, my OWC ThunderBay 6 with Thunderbolt 3--have to be configured properly. I discovered that drives in my striped RAID 0 config should not be in sequential slots. Each pair of slots, A/B, C/D, E/F, share a single PCIe controller. So a three-drive RAID 0 config was *much* faster when in slots A/C/E or B/D/F.

For manual backups, the rsync terminal command (macOS, Linux) is my preferred solution. Copies all/selected files from one drive/RAID to another drive/RAID. New files are added to the destination and existing files that are newer (i.e., modified recently) overwrite the same file on the destination. So your backup will always contain all ("UNION") of the files. rsync doesn't delete files. To save a bit of backup storage I don't backup proxy files. They can be regenerated (somewhat) easily.

And one final note on SSDs. The ultimate in speed would come from a RAID 0 config of M.2 NVMe "drives". Just make sure your computer can handle the data rates!

HTH

For the best of both worlds I run these SSD trays so I can swap out drives easily for projects and databases with the convenience of portable, yet have the performance of internal SATA connection. It works GREAT! I have 4 removable SSD bays.

Jim Simon wrote:System (nothing project related)
Databases
Gallery, Cache, Optimized Media & Proxies
Exports
Media

Jim and Kevin, most new (Windows) boards come with 2 x M.2 ports so assuming not using those for RAID 0 (some of the M.2 disks already do well over 3000 M), what would be an "optimal" distribution of M.2 vs regular SSDs?

Would this make sense:
System (nothing project related) - M.2 (250 G)
Databases - SSD (?? G)
Gallery, Cache, Optimized Media & Proxies - M.2 (1T)
Exports - SSD (?? G)
Media - 2 x SDD in RAID 1 config (or alternatively 2 x spinners if you look for bigger storage)

More concerned about performance while editing, export/render less important.

Being a Mac guy I can't speak to Windows configurations. FWIW, I help folks buy both Mac and Windows--what's best for their applications, wallet (nothing like a cheap Windows laptop if all you want to do is browse the web and use email) and experience.

You're correct that M.2 NVMe disks are quite fast. This can expose the speed of the slot being used, external chassis configuration and connection. So, a disk on the system bus/motherboard can have a more direct connection to the CPU than one in an external enclosure connected with USB-C or (the similar-yet different) Thunderbolt 3. Both might be PCIe but one might be x4 and the other x1. Or, the external enclosure's connector may need to go through separate interface chips before it reaches the system bus. So, lots of variables!

As for SSD speeds, you're loosing capacity for the speed. SSDs seem to max out at 2TB while good hard drives can be 14TB or more. But, you can use RAID 0 to stripe two or more SSDs together, giving you a larger "virtual" volume. Given video's voracious space appetite, having a larger drive can be good. (JBOD with spanning also gives you a larger "virtual" drive.) I don't have specific experience here but if you're dealing with slower interface technology then using RAID 0 could put the NVMes on different interfaces so one doesn't get swamped.

That said, in my limited experience, two things help: 1) putting your video on fast drives and 2) using proxies. So, if you're looking for the best editing experience I'd suggest creating smaller proxies and putting them on your fastest drive. The source video should hopefully be on a fast drive (for generating proxies and delivering). I'm personally less concerned about where my databases are stored.

BTW, you mention RAID 1 which is intended for backup/mirroring. It's always important to have backups, especially of irreplaceable files (aka, source video and edits). You can also use separate backup scripts or solutions. I've heard--but don't know for sure--that having a RAID 1 config can result in some performance degradation. I'm a bit leery of RAID 1 mirroring. If I do something bad and corrupt a file, etc., the corrupted file could be written to both mirrors! So, successfully mirroring failure! With backup scripts I can always go to the backup to recover from the corruption.

Finally, before investing too much, try creating proxies of different resolutions/sizes. See if you can achieve acceptable Resolve editing performance using proxies only. If not, start looking at SSDs and RAIDs.

We are using a G-Speed Shuttle SSD RAID0 for current projects, and it's been fantastic: very fast, very reliable. Because it's a RAID0 (with no easy self-repair), we also have two spinning hard drive backups "just in case."

We were using an external 4TB drive for Gallery/Cache, but just changed to an internal 4TB SSD. Zero speed difference, but it freed up a port. We were relieved that Resolve found the path (identical drive name) with no problem: all stills and caches from past sessions are fine.

For run of the mill sessions, we're using two RAID5 Shuttles, and we just bought a third. Lots of 14TB standalone drives for backup. We're not using proxies at all: we run 5K and 6K Raw all the time, no issues. Heavy OFX do need to be cached, but that's pretty typical. We're looking at networked storage for next year.

We were using an external 4TB drive for Gallery/Cache, but just changed to an internal 4TB SSD. Zero speed difference, but it freed up a port. We were relieved that Resolve found the path (identical drive name) with no problem: all stills and caches from past sessions are fine.

Was the "external 4TB drive" also an SSD? Just curious since you saw no speed difference. At 4TB I assume it also is an SSD.

We're not using proxies at all: we run 5K and 6K Raw all the time, no issues.

I use a 2017 iMac (see signature). With no proxies I can edit one BRAW clip reasonably well, when I add a second parallel clip I start to see stuttering. In both cases reading the BRAW video from an external RAID 0 config. Using proxies, also on the RAID array helps this. That said, I'm more "poseur" than "pro" so config does matter! There might be ways I could speed up my rendering without proxies.