File Handling Preferences

[Peter Chamberlain]

Reposted on behalf of Jason due to system issue.
—-

I've been through most of the PDF training and am still looking for a "best practice" for file handling of things like cache and timelines.

So here are the options that I'm torn between:

1. Using the default Project Library (Local). This stores some project related items on my computer (cache/database).
2. Creating a new Project Library located on my NAS, so all those items are stored alongside my asset files.

I also have cache and renders set to save to an external (USB-3.0 SSD). Should these save here or also on my NAS?

Waaay back when I learned Photoshop, I learned it was at its fastest/most efficient when set up on multiple HDD's: 1 for the OS and program, 1 for the asset files, and 1 as a scratch disk.

Obviously PC's are much faster, but my NAS is the only component with HDD's and not SSD's, meaning it's the slowest part of my system because of spinning disks. Part of me thinks it's best to store everything on the NAS where I have RAID-redundancy (2 mirrored 12-TB HDD's), so if anything goes awry with any 1 drive, it stands to reason I'll always have all my files.
But I set it up such that I am reflecting on my old Photoshop days. The database file on my PC, cache on an external drive, and assets on the NAS.

My computer hits only the minimum requirements for Resolve (it's on a shortlist of upgrades), so I want everything to run as fast and efficiently as possible. I also want to be smart and efficient about all this.

What are your recommendations? Anything about this 3-drive setup that makes sense or doesn't? Am I setting myself up for catastrophe? Looking for advice and guidance. Thanks!

-Jason

Are you on Windows or Mac?

The best practice for most NLEs and Resolve is no exception, is to have the very fastest drives for cache/scratch and current work media locally. In my case that is NVME ssd drives; 1 for scratch/cache/rendering and another for media. I have a big raid setup with spinning HDDs but I try not to work from that; I just offload at the end. These ssds are quite small (2TB) but that is sufficient to online and grade several 1 hour TV projects at once.

In a lot of shared professional environments I work in (Avid/Nexis) all media and projects are obviously stored on the server; these are professionally secure. I would not recommend replicating the same on some home consumer NAS setup; if for nothing else for the speed to which I alluded before.

If you're on Windows (I can't speak for Mac) and you're at the edge of minimum requirements, there are several things that can help with Resolve. Resolve is very GPU intensive/centric app, unlike Avid Media Composer; so there is not a lot you can do if your GPU is very underpowered. But you can leverage Intel for some help by decoding H.26x via Quicksync, which is a choice in preferences. You can also switch off as much of gpu as you can, such as decoding Braw, Red, scopes (though I don't think there's an option for that any longer) etc. to give maximum priority to your limited GPU bandwidth as you can. (those things work fine with just CPU too and especially if you have a lot of ram). Not to mention all the very flexible ways Resolve allows you to lower the burden on original media weight; such as proxies, smart caches, lowering the timeline resolution and all the rest, etc.

I hope this helps and I hope I understood you fully.

Thank you, Steve!
Yes, I am on Windows. Working solo.
My PC SSD is only 500gb, so working there is a no-go (I usually have at least 1 project, usually 2-3 active at any given time, just in different stages). I have an external SSD (NVME), only 256GB that I got as a scratch disk. It's attached to my PC as part of a powered USB hub. I've noticed it throws some errors when my computer wakes from sleep if the drive temporarily disconnects--since moving scratch disks off of this drive, no errors. IDK how much faster it was making things. (I think a lot of my system slowness was related to my lack of knowledge and doing things in ways that weren't helping me, like when I had to re-edit colored footage and didn't know to bypass the color to speed the rendering.)

That leaves my NAS, 2 mirrored spinning HDD's, 12TB each.

I proxy all my footage just to make things run smoother.

I guess my big "end of the day" question is does it matter where the files reside based on how fast the systems are these days?

I'm probably more concerned about project integrity over time, so if I swap out my PC and have to reinstall all my software, what parts of my projects will I lose if things like the project database is stored on my PC rather than the NAS. It starts to make sense to keep everything on my NAS until I have a better system to work with.

I've also been looking at the difference of creating projects versus creating Project Libraries to further isolate where my files are being saved. I so want a definitive "This is the way" answer, but there is none. So I'm experimenting with a 2024 Project Library so every year when I've archived my last project from the previous year, I can delete the Library and all the random associated files that will be recreated if and when I revisit those projects.

Any thoughts/experience you have on this would be appreciated.

The problem with NAS, Jason, though I confess I'm not very familiar with it, is the speed of the ethernet connection; many computers are just 1GB and even 10GB, such as we use with Nexis is only really ideal for HD.

In the end it's less important how fast a drive is; or even if it's raid; than the bandwidth protocol it delivers that data. In your case having an NMVE ssd externally connected via USB, negates any benefits that those drives can deliver.

I'm not sure whether you're on a desktop or laptop. In my case I'm on a high end laptop for reasons of mobility (onsite; offsite); but it still has to deliver the speed and functionality to online professionally, at least in HD for UK broadcast. Therefore I chose NVME ssds that I am able to fit internally and maximise ram and fit the most powerful gpu available. A high end desktop would still be more powerful but it gets the job done for me; and that's the way I planned my setup.

If at all possible it would significantly improve your setup, if you are able to move those ssds to internal. Some of these NVME can achieve speeds up to 20 gigabytes per second (Gbps); which is 3x the speed of even Sata ssds and hugely faster than the best USB connected drive.

As well, and again I maybe wrong here but a USB 3 (or better a thunderbolt) connected raid will be a lot faster than NAS - I can handle HD at full online quality, without issues, from a 2 drive raid connected in this way; though I rarely do.

Again I hope this makes sense and is helpful.

Thank you, Steve.
Yes, the plan is to upgrade the laptop--hopefully later this year. I get the importance of a desktop, but I'm also on the road and have already done the 2-computer thing and am not a fan.

I have a NAS too, but only for backups (and even then, additional ones on single platters). A simple home NAS with HDDs is not great for editing off it, for the reasons Steve already explained. Plus one more: access times.
SSDs are so much better in this respect, even if only connected by USB-3. So, if you can't have an internal NVMe, even simple off-the-shelf SSDs are good for most situations. They'll make your editing experience so much snappier.
Only the software and the Library should reside on your system disk. If the latter ever get's disconnected while at work, you may ruin your whole Library and the automatic backups.
All of this applies to Windows or MacOS alike.

I do all my editing from an external 2T SSD. It means I can move between my Studio Max, iPad or PC. The project is exported to this at the end of each session but of course is also saved on which ever machine is being used at the time. It is formatted exFAT but is reformatted when edit is complete as all the files are also backed up to a RAID 5 anyway. Plenty fast enough for 2 to 4 camera UHD editing.

Jason could you post your PC configuration ?

How will that handle long clips, like in documentary?
I always found file splitting very unreliable. While I can confirm that an external SSD works fine for editing, I’d use NTFS for Windows. If you need to move between PC and Mac, there’s software from Paragon to handle it.

How will that handle long clips, like in documentary?
I always found file splitting very unreliable. While I can confirm that an external SSD works fine for editing, I’d use NTFS for Windows. If you need to move between PC and Mac, there’s software from Paragon to handle it.

All my edits are theatre multicam so close to 1 hour for each act sometimes a little longer. Mix of GH5S UHD 60P h264, GH6 5.7K 60P h265 10bit,and AX100 HD60P most of the time. All single files GH files are .mov with LPCM audio AX100 is XAVC also continuous file. Last show AX files were ACt 1 29G, Act 2 24G, GH5S ACt 1 83G, Act 2 70G, GH6 Act 1 166G, Act 2 144G. So not split files at all. GH5S and AX100 record to 128G SD cards and I tend to use the 512G CFexpress on the GH6. Also record from mic and theatre sound board to a F3 32bit float. So you see all fit easily on the 2T SSD , infact there are two projects on it at the moment with plenty of room left. I am happy with exFAT, will correct my other post getting late here.!!

OK, I reacted while there was still FAT32 written, which really surprised me.
Nevertheless, even if having been bitten only once, I don't trust exFAT either.

My computer (cpu and gpu) is underpowered by today’s standard (see signature) but I am able to handle feature length materials while ingesting intermediate format like ProRes and RAW (BRAW, some ArriRAW, and R3D) up to 4K 60fps and I can attribute it to my setup and some logical planning, which at the top of my list is “NEVER EVER work from a drive that’s connected by wire or wireless connection.” So with that said, I do use a NAS with a RAID 1 configuration (2 x 18Tb 7500rpm NAS drives) and here is my drive setup with DVR studio.

For OS, Resolve Studio, and related apps: 256Gb NVMe on the Motherboard.
For Scratch Drive: 2 x 4Tb 2.5” SSD in RAID 0 configuration. Note: When I will update my MB in the future, I aim to get an MB with multiple M.2 slots and replace the SSD with 2 x 4Tb NVMe and set it up in RAID 0 configuration.
For my work/project asset drive: 2 x 8Tb SSD in RAID 0. This is where I only offload all the assets I am working with at the moment of an active project or two. It’s in RAID 0 as as long I backup my project each day to my NAS, I am not worried of any file corruption on the Work Drive since it is backed up to the NAS.
NAS - Only used to be my stage 1 backup, which is a nightly backup.
Archive drive - I think of this as my stage 2 backup. This is a 5 x 18Tb 3.5 inch slow drive in an external enclosure connected to my PC. At the end of each project, I export a DRA of the project to this drive. After I transfer all the files for a particular project to a client’s external drive (all assets and deliverables), which is taken from the NAS, I delete the project from the NAS and the work drive but keep the DRA on the archive drive for any future adjustments that a client might come back for. I also delete all the scratch files on the scratch drive. This way, I have always room for the next project.

I can do 2-3 90 minutes indie features at a time if I want to with this setup. BTW, I started with a 24Tb NAS and just a 2 x 3Tb HDD in RAID 0 for my work drive and it lasted me a while until clients started bringing large files in 12Tb HDD. There was a time I would just use their external drives to access the assets but it got really painful… which led me to the “NEVER EVER” statement above. Experience lead me to get a 2 x 8Tb SSD and set it up in a RAID 0 because that gave me a bit of a headroom to dump the assets from their 12Tb HDD to the 16Tb RAID 0 SSD, making it pretty fast to work with.

That’s my file handling practice. I don’t know if that can be considered a best practice but that’s what works for me; and I will do the same for my file handling methodology when I upgrade to a newer machine when the right time comes.

My computer (cpu and gpu) is underpowered by today’s standard (see signature) but I am able to handle feature length materials while ingesting intermediate format like ProRes and RAW (BRAW, some ArriRAW, and R3D) up to 4K 60fps and I can attribute it to my setup and some logical planning, which at the top of my list is “NEVER EVER work from a drive that’s connected by wire or wireless connection.” So with that said, I do use a NAS with a RAID 1 configuration (2 x 18Tb 7500rpm NAS drives) and here is my drive setup with DVR studio.

Very interesting Ellory. In a lot of instances, it's not so much a computer being underpowered; as more that media delivery to it can still be the bottleneck. My Zbook 17 G5 laptop is getting near 5 years old now; it was a very expensive bees and knees then but has fallen off even the qualified Avid list now; yet it still does everything professionally I need and to replace it, to an equivalent modern level out of the question financially. Partly because I maxed it out to 128 gig ram and fitted a 16gb gpu card (which are mxm replaceable in these laptops). Some of the advice I gave Jason re Windows settings for Resolve can extract the best with what you've got. This Zbook is still fine for most things with Resolve and more than capable of anything Avid needs, for example.

I personally don't see the need to raid SSDs for speed, certainly not PCIe mounted NVME ones; they are on their own substantially faster than even the fastest Thunderbolt Raid setup you could get. Raid for redundancy might make sense. But as I say I use the NVMEs as the working speed demons for current projects and raid for backup/ storage.

It might be worth touching upon how we do it in and has been done for many years in Broadcast long form. Editing is split, in both jobs and workflow, between 'offline' and 'conform/online/grading' - I do both so I can explain perhaps reliably. It's very rare to work with full quality camera original codecs for offline; this would only ever be done to save time if deadline was very very tight. It is customary to transcode (done by assistants in a machine room at the point of ingest) everything to a working low band codec; such as DNxHR LB; for space and lightness; and to the highest broadcast codec that will be used on final output; say DNxHR HQX; for later conform. Then when the edit is finished, it is conformed onlined and graded and QCed.

I mention all this because it's a useful workflow for anyone; you can see how a 2TB NVMe is sufficient for 1 or more 1 hour TV shows, in this workflow; because in the end I am only dealing with media that has been consolidated from hours and hours of rushes to just 1; in this scenario external raids are more than capable of handing DNxHR LB offline material too.

Whether your pc can handle Arriraw, Red, Braw and full bandwidth Prores, which I'm sure it can, is not a concern in this type of workflow; you are unnecessarily taxing that system's overhead by using it for a 90 minute feature; for the editing process. The workflow I've outlined is also common on Avid for feature films both here and Hollywood too; since I have worked in some of those areas and consulted on workflows for such projects, before.

So personally for feature films what I would do is transcode and offline in a lower bandwidth, from external raids; conform and work in 'reels' (i.e. 10 minute sections' - so you have manageable less taxing chunks also) later at mastering quality codecs from very fast internal NVME ssds, for finishing/fx/colour - all the heavy lifting. In this way whatever machine you are on, you can maximise the resources and overheads of your system. All rendering/caches/proxies still coming from that NVME scratch disk. This is just a suggestion and is the way we work whether it is Avid/Baselight or Resolve.

I just want to thank everyone for the info you're providing. Now that I've "got your attention", I'll go into deeper detail.

I'm new to this. Been in photography since HS (graduated in '89, so it's been a while). Proficient in Photoshop and Lightroom. After a layoff as a project manager for a high volume photography company (school photos), I went to work for/with my wife (she used to shop out the video work to a company that white-labeled her brand on their work). She has a small marketing company that specializes in video storytelling for dentists (think: meet the doc, meet the team, patient testimonial videos). I dove into video, devouring everything I could. Had a professional colorist friend teach me the basics after passing the Resolve user test. That sort of thing--I'm totally Jack of all tradeing this.
I work with aSony FX30 and a6700 as my B-camera. We do all location shooting and need to keep everything to carry-on.

Because of what we do, I'm often working on 2-3 videos at a time (docs are SLOW to approve edits so it can take 1-2 months before I can officially archive a project).

Because of the layoff from a great job and being new, I'm largely working with what I already had, computer-wise.

So.... I guess the best thing I should be asking is:

What do you recommend for working with what I have right now and what do you recommend I work with as I invest in future upgrades?

Currently, this is my setup:

Laptop (specs below)
Connected via USB-C to a powered USB hub/gigabit switch that has:
-power passthrough (to power the PC
-ethernet connection to router and NAS
-256 NVME SSD
-HDMI output for second monitor
-connection to second powered hub with on/off buttons for peripherals (camera, mic, streamdeck, Tangent ripple ball thing for coloring).
My NAS is a Netgear, 2x12TB HDDs.
All on USB battery backup.

• Lenovo IdeaPad 2022
Processor
1x 11th Generation Intel® Core™ i7-1165G7 Processor(Core™ i7-1165G7)
Memory: 1x 16GB DDR4 3200
Operating System: Windows 11 Home 64(EN:English)
Hard Drive: 1x 512GBPCIeGen3x4
Graphics: 1x NVIDIA GeForce MX450 2GB
Monitor: 15.6FHD60Hz
• Wireless Network1x Wireless 2x2 AX; Bluetooth® 5.1 or abovePorts1x HDMI 1.4b; SD Card reader; 1 Type C (USB 3.2 Gen 2 + DP 1.2 + PD 3.0); 2 USB 3.1 (Gen1); Combo of 3.5mm Stereo Headphone Output; DC-in Jack (Round)Camera1x 720P HD with Dual Array Microphone

My current workflow:
After a shoot (while still on location/back at the hotel), I use Resolve to clone all my memory cards to a portable Samsung 1TB SSD.
Once home, I use Resolve to clone the folder on the SSD to my NAS.
I was using the local settings for project database in Resolve (Resolve saves some things to my computer HD), but someone suggested making a new Project Database for every gig (which keeps all the files Resolve usually saves to my computer with the video files on the NAS). I haven't actually done this yet, next shoot is in 1 week from today.
I use the Proxy Generator to create proxy media for all my video files.
I set Resolve Render Cache to Smart. I feel like this is my biggest bottleneck, when it renders.

We're producing 3-4 videos of 2-3 minutes in length, using about 1 hour of footage (per video) from A & B cams, as well as B-roll footage. So there's not a lot actually going on. Most of my work is in the edit and color. There's some lower-third graphics and we create into and outro reels separately. I'm not making movies with any effects.

We are planning to film a short-format documentary this summer. Maybe 20-30 minutes in completed length, similar setups, although instead of 1 filmed subject, we'll have 10-15 that we edit together. But that'll be the most complex job we've ever done.

So... as I mentioned, two questions:
How can I best work with the equipment I have now?

In 6-12 months when I upgrade my laptop and can better handle editing on the internal drives, what else should I invest in and how should I set up that workflow for maximum benefit?

THANK YOU SO MUCH!!!

I work with aSony FX30 and a6700 as my B-camera. We do all location shooting and need to keep everything to carry-on.
…
What do you recommend for working with what I have right now and what do you recommend I work with as I invest in future upgrades?
…
Graphics: 1x NVIDIA GeForce MX450 2GB

So, the first and last point are your biggest issue. That card can hardly handle 'small' HD (1280x720). And even then, you've got no proper hardware decoding for the footage out of the Sonys. The FX30 in particular, a great camera, but years ahead of your computer. Unfortunately, these days any upgrade of the camera is forcing you into computer upgrades too.
I'm afraid, this being a laptop, that GPU can't be upgraded? What you could try for now is converting all your source footage into DNxHD/HR in HD resolution. Try to work in a timeline in 720 and switch to HD only for final render. It may work to some degree for small projects without any demanding functions in DR.
Upgrade? Let me be devil's advocate here: since you want a laptop, get a Mac. Even the cheapest Air, but with 16 GB RAM, will do. But then, in half a year or one year, Windows laptops may have Snapdragons at a similar level or Intel may join again. Right now, Windows PCs are only superior in price/performance on the desktop.

It might be worth touching upon how we do it in and has been done for many years in Broadcast long form. Editing is split, in both jobs and workflow, between 'offline' and 'conform/online/grading' - I do both so I can explain perhaps reliably. It's very rare to work with full quality camera original codecs for offline; this would only ever be done to save time if deadline was very very tight. It is customary to transcode (done by assistants in a machine room at the point of ingest) everything to a working low band codec; such as DNxHR LB; for space and lightness; and to the highest broadcast codec that will be used on final output; say DNxHR HQX; for later conform. Then when the edit is finished, it is conformed onlined and graded and QCed.

I mention all this because it's a useful workflow for anyone; you can see how a 2TB NVMe is sufficient for 1 or more 1 hour TV shows, in this workflow; because in the end I am only dealing with media that has been consolidated from hours and hours of rushes to just 1; in this scenario external raids are more than capable of handing DNxHR LB offline material too.

Whether your pc can handle Arriraw, Red, Braw and full bandwidth Prores, which I'm sure it can, is not a concern in this type of workflow; you are unnecessarily taxing that system's overhead by using it for a 90 minute feature; for the editing process. The workflow I've outlined is also common on Avid for feature films both here and Hollywood too; since I have worked in some of those areas and consulted on workflows for such projects, before.

So personally for feature films what I would do is transcode and offline in a lower bandwidth, from external raids; conform and work in 'reels' (i.e. 10 minute sections' - so you have manageable less taxing chunks also) later at mastering quality codecs from very fast internal NVME ssds, for finishing/fx/colour - all the heavy lifting. In this way whatever machine you are on, you can maximise the resources and overheads of your system. All rendering/caches/proxies still coming from that NVME scratch disk. This is just a suggestion and is the way we work whether it is Avid/Baselight or Resolve.

Steve, thanks for sharing your experience and it’s always great to learn how others do it. I do agree I’m taking the box however I do generate optimized medias using DNxHR LB of the clips I bring in for editing. I don’t have the luxury of an assistant do all the transcoding but it is worth noting.

Thank you all.
The primary question I came here with was: Here's what I have to work with, how can I best optimize my settings to make this work for the time being.

Here's what I'm doing:
Relying on proxies: Slows down color editing a little, but the delay is worth the playback
Working files stay on the NAS: they're just too big for my PC.
All other files are stored locally: Keeping them on my PC SSD means less spinning platters. Since most projects are short-form video that are completed in under two weeks, I can keep these local folders lean.

Eventually I'll upgrade. I don't know whether I'll go desktop or laptop. I understand the upgradability of a desktop, I just hate using two computers in life, and I am constantly using my laptop.

Thank you all.
The primary question I came here with was: Here's what I have to work with, how can I best optimize my settings to make this work for the time being.

Here's what I'm doing:
Relying on proxies: Slows down color editing a little, but the delay is worth the playback
Working files stay on the NAS: they're just too big for my PC.
All other files are stored locally: Keeping them on my PC SSD means less spinning platters. Since most projects are short-form video that are completed in under two weeks, I can keep these local folders lean.

Eventually I'll upgrade. I don't know whether I'll go desktop or laptop. I understand the upgradability of a desktop, I just hate using two computers in life, and I am constantly using my laptop.

I think you’ve got a plan.