Ditch the database approach to managing projects

When I first started working with Resolve, the database approach really caught me off guard. I, as many others, are used to working with a project file that lives on your drive with the other content files used for the project.

I soon adapted and went along my way working with the DB approach. That is until recently.

I have a client project DB that had all the projects i work on for them. Some of those projects are completed and I went to archive then to 'clean up' house. I exported a drp and moved that with the content to my clients archive drive. Then I deleted the project from the DB.

Some sync software I use automatically deleted the drp file from the archive drive (totally my fault for missing that config) and since I deleted it from the DB, there no way for me to recover the project file.

Yes I have backups but the backup is of the DB file which the project is already removed from. I have a recycling bin setup on my NAS so if any files are deleted they sit there for about 30 days before they are gone. Since the project was deleted from within the DB file, that bypasses the recycling bin safety.

By moving to project files, standard backup and versioning solutions can be used to avoid situations like I now find myself in.

I understand that there will be those that prefer the DB approach especially those that work in groups with multiple people in one project. I feel that there should be a choice to work with a flat file method or to start a DB project. Of all software I've used, or still use, Resolve is the only one that has used this approach to project management. If it was a better approach why don't we see this used in other software?

Please don't flame me with "you just don't know what you are doing" or saying this request is stupid and I should just get with the times. This is my opinion and I don't think I'm the only one. If you disagree simply say so or better yet don't post anything. Ignore this and move along. I don't feel like getting into forum fights over opinions and you're not going to change my mind to your way of thinking.

Anyone else feel as I do?

Adrian, you're clearly not stupid and no reasonable person should criticise your request.

I happen to prefer the DB approach as I find it easy/quick to copy them to external drives which gives me the security of backups of not only project files but the whole DB system - I don't understand how DBs work or are constructed and don't need to. I just need them backed-up with everything safe.

To repeat, there's no harm in requesting an alternative method so if any people flame you, just ignore them. They'll get frustrated and find another "victim"!

There is a separate forum for Feature Requests.

viewforum.php?f=33

Best practice is to search first, to see if someone already asked. (In this case, someone has.)

A lot of things like this with Resolve work out to: the more you accept their design philosophy, the faster and easier you can work with it.

We do daily manual backup and versioning of projects, and just save them in a "Color" folder on the source drive. Works great for us. We almost never, ever, ever have to use it, but it's there in the event of a nuclear attack.

The database can be very helpful, particularly in a facility with multiple employees, many different projects, and other kinds of shows that need to be shared in different rooms at the same time. In particular, the PostgreSQL server approach is really flexible and well-done. All of this is covered in the manual in Chapter 3 "Managing Projects and Databases," starting on p. 67. I agree it's not the same approach as some other kinds of software, but a lot of it boils down to finding a way to deal with it and getting it to work for you.

. I exported a drp and moved that with the content to my clients archive drive. Then I deleted the project from the DB.

Some sync software I use automatically deleted the drp file from the archive drive (totally my fault for missing that config) and since I deleted it from the DB, there no way for me to recover the project file.


By moving to project files, standard backup and versioning solutions can be used to avoid situations like I now find myself in.
Anyone else feel as I do?

I am having a hard time trying to understand how a project files format would have saved you when a misconfigured script deletes it to sync the backup.

Seth goldin has an automated database backup tool on GitHub for Resolve .


Sent from my iPad using Tapatalk

the more you accept their design philosophy, the faster and easier you can work with it.

I certainly found that to be true after switching from Premiere Pro. I found the training invaluable.

https://www.blackmagicdesign.com/produc ... ning#books

there no way for me to recover the project file

I'm sorry for your loss, this kind of thing is never a pleasant surprise.

I am having a hard time trying to understand how a project files format would have saved you when a misconfigured script deletes it to sync the backup.

I was thinking the same thing.

The file-based DBs are just... files. One could even use a separate one for each project, in theory.

IMHO a good backups routine includes keeping multiple copies of important files, going back as far as required (not relying on a recycle bin, sorry).

With a PG SQL server it's also very simple to use a one-line scheduled script[1] to back up the database(s) with unique names.

Restore an older DB (file or server), connect it to Resolve, find deleted project(s). Only 1 extra step vs. using project files directly.

Also, there are Resolve API scripts out there to export projects to files automatically or on demand, which is also a one-line operation (ExportProject(projectName, filePath, withStillsAndLUTs=True)). Make the filePath dynamic with a current date/time and you have versioned backups.

Not to mentioned the automatic project backup feature already provided by Resolve.

Cheers,
-Max

[1] One-line database backup script with daily versioned file names. Also see pg_dump.

Code: Select all

Linux/MacOS:
pg_dump -U postgres resolve > /Backups/pgsql/$(date "+%Y%m%d").resolve.bak.sql
Windows:
pg_dump -U postgres resolve > c:/Backups/pgsql/%date:~10,4%%date:~4,2%%date:~7,2%.resolve.bak.sql


If the DB was diskdb and not a PostgreSQL DB, you could try restoring the DB file itself if you have file history on.

As with anything, you'd be best to make backups just in case

I'll try to explain how working with a DB rather than a project file has caveats.

A Resolve DB file can contain multiple projects in it. A backup task will copy those DB files. I backup onto a NAS which then backs up to another NAS. Deleting a project from within the DB doesn't get caught by the recycle bin on any system as the backed up files are still there, they simply changed. So the back up kicks in and overwrites those backup copies. There's no way to restore anything deleted from within that isolated file structure.

My project file mistakenly was deleted as the archive drive I was moving the exported project to was an external USB drive (no recycle bin on that drive). A common approach to archiving projects. If I had been working with a premiere project file I would have had two copies of the file sitting in two different recycle bins on both NAS backup volumes.

I don't see why it is such a difficult thing. Resolve already exports drp files so it has a project file format support already. Why not give us the choice to work in a drp file based approach or a DB approach when we create a new project?

Thinking about it, I don't think there is even one other application I've worked with that uses the approach resolve uses. In all my years, I can't recall one.

A Resolve DB file can contain multiple projects in it. A backup task will copy those DB files. I backup onto a NAS which then backs up to another NAS. Deleting a project from within the DB doesn't get caught by the recycle bin on any system as the backed up files are still there, they simply changed. So the back up kicks in and overwrites those backup copies. There's no way to restore anything deleted from within that isolated file structure.

I still don't get how this is different from any other file. Sorry. Say I open a Word document, delete a bunch of text and then save it. The contents have changed, but the file still exists. W/out a versioned backup, after your backup cycle, the deleted text is gone forever. After my backup cycle, I can go back and get the original document from up to a year ago (maybe longer if it was a very important file).

Anyway as I said there are other options to back up individual projects automatically. Including simply using one DB file per project. But you did say your mind is already set on this.

Cheers,
-Max

My project file mistakenly was deleted as the archive drive I was moving the exported project to was an external USB drive (no recycle bin on that drive). A common approach to archiving projects. If I had been working with a premiere project file I would have had two copies of the file sitting in two different recycle bins on both NAS backup volumes.

We don't delete anything, ever, at least not related to a DaVinci Resolve Project file per se. I have files going back 9-10 years for sure. but they're in unactivated databases that we've disconnected. The project files still survive in backups.

Two keys: 1) don't delete anything (by mistake or otherwise), and 2) any renaming or moving or copying of projects pretty much has to be done within the Project Manager. External disk databases are problematic because of the risk of momentary unmounting, which all drives can do under some conditions. For any colorist, editor, or facility that makes a living with Resolve, I'd suggest going with a PosgreSQL database, which is inexpensive, not that hard to set up, and infinitely more flexible. Those can use external drives.

Backups will save you in crisis situations. I've had to use backup sessions twice when a client's system failed, and once when I had a session get corrupted on my own system and some work on a piece of a timeline got wiped up. In my case, it was only 3-4 hours of work, but that was a rare situation where the session had stopped saving and I didn't notice. Disk, PosgreSQL, or your suggested file-based session would not have helped if it wasn't saving.

We do daily backups for all current sessions, and in some cases use additional Cloud-based storage just in the unlikely event the system or the drive failed. That hasn't happened yet, but it's good to know the backups are there. The DRPs are such small files, it's fairly trivial to save them; once a project is done, I generally only save what I call the final project, which would be like "Client_name_Project_Name_FINAL_6-27-2021.drp", something like that. For projects done over a week or two, we keep each day's work, which helps determine billing: I can glance over and say, "ah, we worked on that for six days, so that's X hours for that project."

This won't give you a file-based alternative to the Project Database, but it will give you backup in the unlikely case the database fails or the original session file gets deleted. This falls under the category of "best practices," and it's probably a good topic for a tutorial someday.

Regardless if you use project files , db files , postgresql or a combination, or even with what program you work as this is not Reolve specific, your backup strategy needs to be solid, consisting of min 3 diff locations and keeping versions back to how ever long you need on each location (can differ per location based on requirements). And of course fully tested/verified regularly
Your current strategy, if we can call it that, is sorry to say not the best and a backup strategy that relies on a recycle bin (first i ever hear of such) or keeping only a backup copy(s) of the current active db and not older versions (which would have solved your issue) is not a backup strategy.
So do some reading , set it up properly and learn from this experience to prevent it from happening again.

Its always incredibly painfull to learn the hard way and i am sorry that this happend to you. Surely we all somehow faced it someway in our careers that famous 'crap' moment, but give it a positive swing and learn from it .

p.s. and this is not trying to change your mind , which you indicate you wont) but to provide you with proper info to make the right choices going forward. You are of course free to ignore and do it your way , but you just found out the effects of doing it your way.

I've looked at Seth's github for backups. It looks like it's something we can once as we migrate from Premiere. One of the other things I was wondering is how easy is it to migrate from one DB to another through Resolve? We have active projects on our SAN. Once they are completed, we move to an Archive network location for a year. Then those projects are migrated completely offline to tape.

What's the best way to achieve this with Resolve, or does our workflow need to change? Do we export individual DRPs from the Archive DB and back those up? Or is it better to say after a year, whatever is in that archive DB is archived and we start with a new Archive DB?

Theoretically, how many projects can reside in a DB (thinking archive) before it craps out?

It's as easy as selecting the relevant projects in one DB and Ctrl-C

Then Ctrl-V into the new DB

It's as easy as selecting the relevant projects in one DB and Ctrl-C

Then Ctrl-V into the new DB

But it could (and should) be made even easier.

viewtopic.php?f=33&t=96952