Project Archive - Saving Cache to save VFX Connect clips?

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

Steve Alexander

  • Posts: 4517
  • Joined: Mon Mar 23, 2015 2:15 am

Project Archive - Saving Cache to save VFX Connect clips?

PostSat Dec 28, 2019 4:53 pm

Hi folks,

I wondered if there is a way to save the VFX Connect clips (those created inside Resolve but open to the standalone Fusion Studio application) as part of the project archive so that they can be backed up and restored either to the original workstation or to another workstation. I have included cached files option in the archive and it would appear that this also saves the VFX Connect clips but upon restoring from the archive the VFX Connect clips are not correctly restored.

Is there a prescribed way to do this? Otherwise the VFX Connect feature is difficult to use.

Resolve Studio 16.1.2

Thanks in advance
Time Traveller
Resolve Studio 18.6.6 | Fusion Studio 18.6.6 | Win 11 Pro (22H2) | i9-7940x, P4000 (536.96, 8GB VRAM), 64GB RAM, M.2 boot, SSD scratch, RAID10 data | (laptop) 16" MacBook Pro M1 MAX, 32 GPU cores, 64 GB RAM, 2 TB SSD, Sonoma 14.4
Offline

Dermot Shane

  • Posts: 2720
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: Project Archive - Saving Cache to save VFX Connect clips

PostSat Dec 28, 2019 5:15 pm

resolve archive, keep caches

another option is to save the database, and restore that with the meida in excatly the same place as the orignal project

archive is simpler
Offline

Steve Alexander

  • Posts: 4517
  • Joined: Mon Mar 23, 2015 2:15 am

Re: Project Archive - Saving Cache to save VFX Connect clips

PostSat Dec 28, 2019 6:47 pm

Thanks Dermot, but as you will note in my original post, the 'keep caches' does not restore the VFX connect project correctly. This may be a bug in a recent release but I have not tried to do this before so I can't say. Thanks for confirming the workflow that is at least supposed to work.

Cheers
Time Traveller
Resolve Studio 18.6.6 | Fusion Studio 18.6.6 | Win 11 Pro (22H2) | i9-7940x, P4000 (536.96, 8GB VRAM), 64GB RAM, M.2 boot, SSD scratch, RAID10 data | (laptop) 16" MacBook Pro M1 MAX, 32 GPU cores, 64 GB RAM, 2 TB SSD, Sonoma 14.4
Offline

Sean Weaver

  • Posts: 184
  • Joined: Fri May 21, 2021 6:37 am
  • Real Name: Sean Weaver

Re: Project Archive - Saving Cache to save VFX Connect clips

PostThu Oct 21, 2021 12:19 am

Dermot Shane wrote:resolve archive, keep caches

another option is to save the database, and restore that with the meida in excatly the same place as the orignal project

archive is simpler


Found this thread via a search of the forums. Can anyone expand upon this quote above please? What I am trying to setup is an idiot-proof level of security with off-site backups containing .DRA archives of source media, and Fusion Studio compositions via VFX Connect. It's unlikely anything this catastrophic would happen but I want a way where if, for example, the house flooded and all my drives were toasted, I would just have to get my off-site drive, restore the Project Archive (containing VFX Connect clips), make sure the composition file (which Export Project Archive does not do by default) is present, and load everything up. I have spent several hours today deliberately creating sample archives and attempting to play this out in a hypothetical scenario, but so far, my VFX Connect has not linked up. I can see a situation where as long as the .DRA and the composite render .mov are present, I could put footage back together...but I'm really very interested in solving how to retain full access to the original Fusion Studio composition (via VFX Connect) along with the restoration of the Project Archive. I even tried what the above quote says, but am not sure I understood and in the end it did not link up with a database restored to the same folder as the source footage in the archive folder. To test this out, I am ejecting my main work SSD Raid drive and restoring archives from other external drives or the internal Mac drive. What I am looking for is a way to be secure if, for example, the original source footage drive (or just volume) no longer exists.

The VFX Connect clips link up flawlessly every time when the main work drive is connected, but as soon as I eject the drive and run it through this hypothetical disaster recovery scenario, the integration falls apart and I'm left without a clip that can be opened back up in Fusion Studio.

The reason I am using VFX Connect instead of working entirely within Resolve is my computer can handle the footage using stand alone, whereas in Resolve, I just don't have quite enough computer. That, and that I am able to import and audio waveform and work faster/better on keyframing to music in Fusion stand alone

Advice?
DR 18 Studio
macOS Monterey 12.7.2
MBP 16-Inch, 2019
2.4 GHz 8-Core Intel i9
32GB 2667 MHz DDR4
AMD Radeon Pro 5600M 8GB
Offline
User avatar

TheBloke

  • Posts: 1905
  • Joined: Sat Nov 02, 2019 11:49 pm
  • Location: UK
  • Real Name: Tom Jobbins

Re: Project Archive - Saving Cache to save VFX Connect clips

PostThu Oct 21, 2021 10:40 am

DRA Archives and VFX Connect

Yes, VFX Connect is not fully handled in a DRA.

If, when creating the VFX Connect, you choose a Custom Location, it will never be included in the DRA.

If you do not choose a custom location - meaning it goes into the CacheClip folder - it will be included, and restored by the DRA, if "Cache Files" is selected when making the DRA.

However it will not then automatically reconnect when the DRA is imported. Upon importing the DRA and looking at a VFX Connect clip on a timeline, you will see the original clip, not any Fusion-rendered version. If you then choose to "Open In Fusion", it will re-render the media out of Resolve, and load up a blank Fusion composition.

This appears to be a bug or an oversight.

However it is possible, with a couple of manual steps, to restore a DRA and then move in the restored Fusion composition and any previous Fusion renders.

The heart of the problem is that the VFX Connect - like all cached media - is stored in folders named with Resolve database IDs:
Image

These folders are backed up and restored through DRAs, as long as they are part of the standard CacheClip folder configured in Project Settings.

However, upon restoring the DRA, these database IDs change. The database IDs related to the VFX Connect Clips created when you import from the DRA are not the same as they were in the original project. Therefore any VFX Connect clips imported from the DRA are not seen by Resolve unless you fix this manually.

DRPs work

This DRA problem does not happen with a simple DRP export. If you export a project as a DRP, then import that DRP, the database IDs are still correct. All VFX Connect clips will work immediately - regardless of whether they're in CacheClip, or a custom location.

This means that a DRP combined with a generic disk backup should work fine.

How To Make And Restore A DRA Containing VFX Connect Clips

1 To make this process a little easier, it's recommended to configure Project Settings with a CacheClip folder specific to this project:
Image

2 When creating the VFX Connect clips, do not click "Custom Location". Your VFX Connect clips must be stored in the CacheClip folder along with Render Cache.

3 When creating the DRA, make sure "Cache Files" is ticked

4 Restore the DRA as normal

5 In Finder/Explorer, go to your CacheClip folder - the unique one configured in Project Settings - and notice that, upon restoring, it has created a new folder under CacheClip with a database ID. This Database ID is wrong for the newly imported project. Delete this folder

6 In Finder/Explorer, find your original DRA and open it - it's just a folder (at least on macOS it is). Find the Compositions folder.

7 In Resolve, open the newly imported project.

8 In Resolve, find one VFX Connect Clip and put the playhead over it. Notice that it is not showing any Fusion Studio rendered media, it shows the original Resolve media. The VFX Connect has not yet been properly restored.

9 Right-click on a VFX Connect Clip and choose "Render media". It'll take some time to re-render the Resolve media for this clip - so it'd make sense to do this step on your shortest clip, ideally one with no effects contained (Resolve effects that is - doesn't matter what the Fusion Studio comp does). We're going to delete it immediately, we just need to do this once to get the correct database ID for the newly imported project.

10 Go back to the CacheClip folder, and notice a new folder has been created, under which is a Compositions folder. Let's say this folder is d5071cdb-71b6-44a4-aae0-bd81e7303565.

11 Delete the 'Compositions' folder you found in step 10 (under d5071cdb-...) and all its contents. Do not delete the main folder (d5071cdb-...) itself.

12. Drag the Compositions folder from the DRA, as found in step 6, under the CacheClip folder from step 10 (d5071cdb-...)

13 Go back to Resolve, and move your playhead to another VFX Connect Clip. You should now see your Fusion Studio-rendered media. The media is now fully restored.

14 Right-clicking on any VFX Connect Clip should now show all the correct options, and Open In Fusion should find and open your original, restored Fusion composition, including any changes you made to it prior to making the DRA.

I know that's a bit complicated, so here's a quick and dirty video demonstration:


Backup methods

Should you use the above method? You could, it's not that tedious - especially given it's not going to be done often, or hopefully at all.

But no, personally I wouldn't even bother.

I would just implement backups that cover both the project database and all the media locations, including the folder where the VFX Connect clips are being written to/read from. That could be your CacheClip folder or, if you don't want to back up Render Cache, choose the Custom Location option when making your VFX Connects and include that location in your backup, but not CacheClip.

If you back up all that, then upon doing a full restore from a backup, you can't fail to put everything back the way it was and everything will work.

I think that's the point Dermot was making: if you restore your VFX Connect clips to the same location they're at now, then it will all work automatically.

I use Carbon Copy Cloner on macOS to do my backups. It can backup to another drive, and/or to a disk image, which could be on another local drive or a network drive. It's much more configurable and sophisticated than Time Machine. It also allows for making fully bootable backups.

If ever I lost all my drives, and assuming my backups weren't also affected, I could restore to the exact point I was at as of the last backup. No steps would be required other than restoring the backup, because it would include my entire database, all media, etc.
Resolve Studio 17.4.3 and Fusion Studio 17.4.3 on macOS 11.6.1

Hackintosh:: X299, Intel i9-10980XE, 128GB DDR4, AMD 6900XT 16GB
Monitors: 1 x 3840x2160 & 3 x 1920x1200
Disk: 2TB NVMe + 4TB RAID0 NVMe; NAS: 36TB RAID6
BMD Speed Editor
Offline

Sean Weaver

  • Posts: 184
  • Joined: Fri May 21, 2021 6:37 am
  • Real Name: Sean Weaver

Re: Project Archive - Saving Cache to save VFX Connect clips

PostFri Oct 22, 2021 3:52 am

TheBloke wrote:DRA Archives and VFX Connect
I use Carbon Copy Cloner on macOS to do my backups. It can backup to another drive, and/or to a disk image, which could be on another local drive or a network drive. It's much more configurable and sophisticated than Time Machine. It also allows for making fully bootable backups.


Awesome. Thank you for all of the above. It helps tremendously. I switched computers at the top of 2021 and forgot that I own the licensed version of Carbon Copy Cloner 5 as well (though now I see there is a new version).

I'll be giving this a go this evening!
DR 18 Studio
macOS Monterey 12.7.2
MBP 16-Inch, 2019
2.4 GHz 8-Core Intel i9
32GB 2667 MHz DDR4
AMD Radeon Pro 5600M 8GB
Offline

Sean Weaver

  • Posts: 184
  • Joined: Fri May 21, 2021 6:37 am
  • Real Name: Sean Weaver

Re: Project Archive - Saving Cache to save VFX Connect clips

PostFri Oct 22, 2021 6:44 am

TheBloke wrote:
I think that's the point Dermot was making: if you restore your VFX Connect clips to the same location they're at now, then it will all work automatically.


Thanks again for all the help, Tom. I have an update, question, story (I'll make it as short as the events can be told), and great relief.

First, I have 8 videos already done and the VFX Connect clips were not created in the Cache folder, so I had to follow your Carbon Copy Cloner suggestion to the letter for at least these first several. In the process, I have been looking for a way to safely simulate disaster without actually creating one. (verdict: successfully completed but not without a little stress).

I got CCC up and running, connected my backup drive, and picked a single project folder to clone, along with the database. But upon restoring, things weren't linked up right, so I tried it to another disk. Still not right.

This is a little bit embarrassing to show just how much I have had to learn, but along the way, it occurred to me that the file paths are all connected to the disk volume of the original media. So therefore, if the original files were on "SSD Mirror" and I cloned it to "Off Site Backup," then restoring to the local hard drive ("seanweaver") was of course not going to provide the same path, so I tried something else....

Not wanting to take on the stress of renaming any disk volumes with data on it, I went about this simulation another way, as I'm running an 8-disk drive enclosure with SoftRaid for RAID/disk volume creation and management. Of those 8-disks, I still had two containing absolutely no data. Very carefully, I ejected all of the volumes containing any media, and created a new small disk volume on one of the blank disks. I named this volume "SSD Mirror."

I then restored the backup to that new volume, restored the test backup database, and went to open the project (containing the VFX Connect file) that had been cloned in the simulation. Everything loaded up flawlessly and when I went to the VFX file to select "Open In Fusion," it opened in Fusion Studio with everything intact and online. Success.

At this stage I disconnected the "new" "simulate a disaster but try not to cause one" volume and remounted my main work drive (the true "SSD Mirror" containing much more data). Going back into Resolve and selecting the original database, I loaded a project. "Media Offline." (at this point I started to sweat). I then loaded a second project from the same database. It too said "Media Offline." I tried "Relink Media" but the VFX Connect Clip still wouldn't link back up and when I went to "open in Fusion," it generated an error: "Unable to create folder /Volumes/SSD Mirror/BMPCC Footage....(the file path continued in the notification).

I'm still not sure why the next part happened, because I had only tried simulating disaster with a single project. I thought I should check the other projects in the database and so opening the next one, it too greeted me with "Media Offline." This project had never contained a VFX Connect Clip but had always contained a standard Fusion Clip (working entirely within Resolve). The remaining 6 projects were fine and loaded normally, with everything online and just as I had saved them.

I thought to examine this closer in the Media Pool, and so when I did, I noticed that the source footage file path was now reading "SSD Mirror 1" (on with the rest of the file path after that) instead of "SSD Mirror." I selected Change Source Folder and redirected the offline clips to my main work drive (the only one mounted/connected at the time), and thankfully everything came back online and linked up. I checked the VFX Connect Clip and selected "Open In Fusion." This time it loaded like a champ and I was back up and running. Moving onto the second project that had gotten wonky, the Media Pool revealed that it too was looking for "SSD Mirror 1" in the file path, and so I selected Change Source Folder again and pointed it back to the correct location. With my media back online, after a quick restart, the Fusion Clip (within Resolve) no longer said "offline" either. Everything is back to normal. I've opened every project and checked everything with a bit of OCD tendencies. Thankfully I didn't cause a disaster in attempting to simulate one.

The only thing I wonder in hindsight is if maybe I had not disconnected the test volume before loading up the original database again. It was a little stressful and I was a little nervous about it all so I just don't remember. Maybe if I did that, then that's what changed the file path in my regular database. After all, I would have had two identically named volumes connected.

In any event, stressful as it was, I did confirm that in the event of losing a drive, the house flooding, an earthquake, or whatever unforeseen catastrophe, if I just use Carbon Copy Cloner and store the drive off-site, then this definitely will work as a recovery method with VFX Connect clips loading correctly in Fusion Studio.

Now for the actual Carbon Copy Clone process. I don't need to backup the entire drive or even close to it. Just the folders containing the source media and database.

Couldn't have done this without you and can't thank you enough. You're literally helping me get the ball rolling on a project over 3-years in the making. With the video now beginning and ramping up, I realized that now is the time it would become a catastrophe to get any further into this without a tested, reliable, fail-proof backup system and protocol in place. The idea is that at any point in time, if I lose 3 hours of work, fine. Losing 3-week's (or a year's) could sink me. I've given everything to developing this product and taken the better part of a year off from other work to invest in myself to learn all of this and get it off the ground. Obviously now is the time to get my sh*t together (in the street vernacular) with backup protocols and knowing exactly how it would play out if I lost all the drives in the house (stranger things have happened; in 2010 I lost a bunch of guitars and amps in the Nashville floods) and had to restore from an off-site backup.

I do wish there was a viable Cloud solution that I could practically implement. I just don't have the bandwidth or upload speeds with .braw footage
DR 18 Studio
macOS Monterey 12.7.2
MBP 16-Inch, 2019
2.4 GHz 8-Core Intel i9
32GB 2667 MHz DDR4
AMD Radeon Pro 5600M 8GB

Return to DaVinci Resolve

Who is online

Users browsing this forum: RedRider14 and 140 guests