Ditch the database approach to managing projects

  • Author
  • Message
Offline
User avatar

aglyons

  • Posts: 232
  • Joined: Sat Dec 08, 2018 4:19 am
  • Real Name: Adrian Lyons

Ditch the database approach to managing projects

PostFri Jun 25, 2021 2:25 pm

When I first started working with Resolve, the database approach really caught me off guard. I, as many others, are used to working with a project file that lives on your drive with the other content files used for the project.

I soon adapted and went along my way working with the DB approach. That is until recently.

I have a client project DB that had all the projects i work on for them. Some of those projects are completed and I went to archive then to 'clean up' house. I exported a drp and moved that with the content to my clients archive drive. Then I deleted the project from the DB.

Some sync software I use automatically deleted the drp file from the archive drive (totally my fault for missing that config) and since I deleted it from the DB, there no way for me to recover the project file.

Yes I have backups but the backup is of the DB file which the project is already removed from. I have a recycling bin setup on my NAS so if any files are deleted they sit there for about 30 days before they are gone. Since the project was deleted from within the DB file, that bypasses the recycling bin safety.

By moving to project files, standard backup and versioning solutions can be used to avoid situations like I now find myself in.

I understand that there will be those that prefer the DB approach especially those that work in groups with multiple people in one project. I feel that there should be a choice to work with a flat file method or to start a DB project. Of all software I've used, or still use, Resolve is the only one that has used this approach to project management. If it was a better approach why don't we see this used in other software?

Please don't flame me with "you just don't know what you are doing" or saying this request is stupid and I should just get with the times. This is my opinion and I don't think I'm the only one. If you disagree simply say so or better yet don't post anything. Ignore this and move along. I don't feel like getting into forum fights over opinions and you're not going to change my mind to your way of thinking.

Anyone else feel as I do?
Offline

Stewart Hemley

  • Posts: 80
  • Joined: Sat Apr 25, 2015 7:00 pm
  • Location: UK, Japan

Re: REQ:Ditch the database approach to managing projects

PostFri Jun 25, 2021 3:39 pm

Adrian, you're clearly not stupid and no reasonable person should criticise your request.

I happen to prefer the DB approach as I find it easy/quick to copy them to external drives which gives me the security of backups of not only project files but the whole DB system - I don't understand how DBs work or are constructed and don't need to. I just need them backed-up with everything safe.

To repeat, there's no harm in requesting an alternative method so if any people flame you, just ignore them. They'll get frustrated and find another "victim"!
Offline

Jim Simon

  • Posts: 17695
  • Joined: Fri Dec 23, 2016 1:47 am

Re: REQ:Ditch the database approach to managing projects

PostFri Jun 25, 2021 5:00 pm

There is a separate forum for Feature Requests.

viewforum.php?f=33

Best practice is to search first, to see if someone already asked. (In this case, someone has.)
Offline
User avatar

Marc Wielage

  • Posts: 7581
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: REQ:Ditch the database approach to managing projects

PostSat Jun 26, 2021 12:55 am

A lot of things like this with Resolve work out to: the more you accept their design philosophy, the faster and easier you can work with it.

We do daily manual backup and versioning of projects, and just save them in a "Color" folder on the source drive. Works great for us. We almost never, ever, ever have to use it, but it's there in the event of a nuclear attack.

The database can be very helpful, particularly in a facility with multiple employees, many different projects, and other kinds of shows that need to be shared in different rooms at the same time. In particular, the PostgreSQL server approach is really flexible and well-done. All of this is covered in the manual in Chapter 3 "Managing Projects and Databases," starting on p. 67. I agree it's not the same approach as some other kinds of software, but a lot of it boils down to finding a way to deal with it and getting it to work for you.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

G0bble

  • Posts: 291
  • Joined: Wed Nov 21, 2018 10:04 am
  • Real Name: Rahul Sawarkar

Re: REQ:Ditch the database approach to managing projects

PostSat Jun 26, 2021 7:54 am

aglyons wrote:. I exported a drp and moved that with the content to my clients archive drive. Then I deleted the project from the DB.

Some sync software I use automatically deleted the drp file from the archive drive (totally my fault for missing that config) and since I deleted it from the DB, there no way for me to recover the project file.


By moving to project files, standard backup and versioning solutions can be used to avoid situations like I now find myself in.
Anyone else feel as I do?


I am having a hard time trying to understand how a project files format would have saved you when a misconfigured script deletes it to sync the backup.
DVR Studio
Win 10 Pro, Ryzen 3900X, Nvidia RTX 2070,32GB 3200Mhz RAM, SM951 M.2 AHCI SSD Boot, 512GB WD Blue nvme scratch
Offline

Jason Conrad

  • Posts: 719
  • Joined: Wed Aug 16, 2017 3:23 pm

REQ:Ditch the database approach to managing projects

PostSat Jun 26, 2021 8:03 am

Seth goldin has an automated database backup tool on GitHub for Resolve .


Sent from my iPad using Tapatalk
-MacBook Pro (14,3) i7 2.9 GHz 16 GB, Intel 630, AMD 560 x1
-[DR 17.0 Beta9]
Offline

Jim Simon

  • Posts: 17695
  • Joined: Fri Dec 23, 2016 1:47 am

Re: REQ:Ditch the database approach to managing projects

PostSat Jun 26, 2021 6:45 pm

Marc Wielage wrote:the more you accept their design philosophy, the faster and easier you can work with it.
I certainly found that to be true after switching from Premiere Pro. I found the training invaluable.

https://www.blackmagicdesign.com/produc ... ning#books
Offline
User avatar

Max Paperno

  • Posts: 64
  • Joined: Sun May 02, 2021 12:49 am
  • Location: Upstate NY, US
  • Real Name: Max Paperno

Re: REQ:Ditch the database approach to managing projects

PostSat Jun 26, 2021 8:33 pm

aglyons wrote:there no way for me to recover the project file

I'm sorry for your loss, this kind of thing is never a pleasant surprise. :oops:

G0bble wrote:I am having a hard time trying to understand how a project files format would have saved you when a misconfigured script deletes it to sync the backup.

I was thinking the same thing.

The file-based DBs are just... files. One could even use a separate one for each project, in theory.

IMHO a good backups routine includes keeping multiple copies of important files, going back as far as required (not relying on a recycle bin, sorry).

With a PG SQL server it's also very simple to use a one-line scheduled script[1] to back up the database(s) with unique names.

Restore an older DB (file or server), connect it to Resolve, find deleted project(s). Only 1 extra step vs. using project files directly.

Also, there are Resolve API scripts out there to export projects to files automatically or on demand, which is also a one-line operation (ExportProject(projectName, filePath, withStillsAndLUTs=True)). Make the filePath dynamic with a current date/time and you have versioned backups.

Not to mentioned the automatic project backup feature already provided by Resolve.

Cheers,
-Max

[1] One-line database backup script with daily versioned file names. Also see pg_dump.
Code: Select all
Linux/MacOS:
pg_dump -U postgres resolve > /Backups/pgsql/$(date "+%Y%m%d").resolve.bak.sql
Windows:
pg_dump -U postgres resolve > c:/Backups/pgsql/%date:~10,4%%date:~4,2%%date:~7,2%.resolve.bak.sql
Offline

RikshaDriver

  • Posts: 435
  • Joined: Sun Aug 12, 2018 10:08 am
  • Location: Melbourne
  • Real Name: Asim Siddiqui

Re: REQ:Ditch the database approach to managing projects

PostSun Jun 27, 2021 12:09 am

If the DB was diskdb and not a PostgreSQL DB, you could try restoring the DB file itself if you have file history on.

As with anything, you'd be best to make backups just in case
Resolve Looks, Plugins & Transforms: https://xtremestuff.net/store
Open Source Resolve transforms: https://github.com/xtremestuff
Offline
User avatar

aglyons

  • Posts: 232
  • Joined: Sat Dec 08, 2018 4:19 am
  • Real Name: Adrian Lyons

Re: REQ:Ditch the database approach to managing projects

PostSun Jun 27, 2021 3:16 am

I'll try to explain how working with a DB rather than a project file has caveats.

A Resolve DB file can contain multiple projects in it. A backup task will copy those DB files. I backup onto a NAS which then backs up to another NAS. Deleting a project from within the DB doesn't get caught by the recycle bin on any system as the backed up files are still there, they simply changed. So the back up kicks in and overwrites those backup copies. There's no way to restore anything deleted from within that isolated file structure.

My project file mistakenly was deleted as the archive drive I was moving the exported project to was an external USB drive (no recycle bin on that drive). A common approach to archiving projects. If I had been working with a premiere project file I would have had two copies of the file sitting in two different recycle bins on both NAS backup volumes.

I don't see why it is such a difficult thing. Resolve already exports drp files so it has a project file format support already. Why not give us the choice to work in a drp file based approach or a DB approach when we create a new project?

Thinking about it, I don't think there is even one other application I've worked with that uses the approach resolve uses. In all my years, I can't recall one.
Offline
User avatar

Max Paperno

  • Posts: 64
  • Joined: Sun May 02, 2021 12:49 am
  • Location: Upstate NY, US
  • Real Name: Max Paperno

Re: REQ:Ditch the database approach to managing projects

PostSun Jun 27, 2021 3:29 am

aglyons wrote:A Resolve DB file can contain multiple projects in it. A backup task will copy those DB files. I backup onto a NAS which then backs up to another NAS. Deleting a project from within the DB doesn't get caught by the recycle bin on any system as the backed up files are still there, they simply changed. So the back up kicks in and overwrites those backup copies. There's no way to restore anything deleted from within that isolated file structure.

I still don't get how this is different from any other file. Sorry. Say I open a Word document, delete a bunch of text and then save it. The contents have changed, but the file still exists. W/out a versioned backup, after your backup cycle, the deleted text is gone forever. After my backup cycle, I can go back and get the original document from up to a year ago (maybe longer if it was a very important file).

Anyway as I said there are other options to back up individual projects automatically. Including simply using one DB file per project. But you did say your mind is already set on this.

Cheers,
-Max
Offline
User avatar

Marc Wielage

  • Posts: 7581
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: REQ:Ditch the database approach to managing projects

PostMon Jun 28, 2021 3:48 am

aglyons wrote:My project file mistakenly was deleted as the archive drive I was moving the exported project to was an external USB drive (no recycle bin on that drive). A common approach to archiving projects. If I had been working with a premiere project file I would have had two copies of the file sitting in two different recycle bins on both NAS backup volumes.

We don't delete anything, ever, at least not related to a DaVinci Resolve Project file per se. I have files going back 9-10 years for sure. but they're in unactivated databases that we've disconnected. The project files still survive in backups.

Two keys: 1) don't delete anything (by mistake or otherwise), and 2) any renaming or moving or copying of projects pretty much has to be done within the Project Manager. External disk databases are problematic because of the risk of momentary unmounting, which all drives can do under some conditions. For any colorist, editor, or facility that makes a living with Resolve, I'd suggest going with a PosgreSQL database, which is inexpensive, not that hard to set up, and infinitely more flexible. Those can use external drives.

Backups will save you in crisis situations. I've had to use backup sessions twice when a client's system failed, and once when I had a session get corrupted on my own system and some work on a piece of a timeline got wiped up. In my case, it was only 3-4 hours of work, but that was a rare situation where the session had stopped saving and I didn't notice. Disk, PosgreSQL, or your suggested file-based session would not have helped if it wasn't saving.

We do daily backups for all current sessions, and in some cases use additional Cloud-based storage just in the unlikely event the system or the drive failed. That hasn't happened yet, but it's good to know the backups are there. The DRPs are such small files, it's fairly trivial to save them; once a project is done, I generally only save what I call the final project, which would be like "Client_name_Project_Name_FINAL_6-27-2021.drp", something like that. For projects done over a week or two, we keep each day's work, which helps determine billing: I can glance over and say, "ah, we worked on that for six days, so that's X hours for that project."

This won't give you a file-based alternative to the Project Database, but it will give you backup in the unlikely case the database fails or the original session file gets deleted. This falls under the category of "best practices," and it's probably a good topic for a tutorial someday.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline
User avatar

Glenn Venghaus

  • Posts: 1303
  • Joined: Wed Jan 01, 2014 9:56 pm
  • Location: Amsterdam , The Netherlands

Ditch the database approach to managing projects

PostMon Jun 28, 2021 4:59 am

Regardless if you use project files , db files , postgresql or a combination, or even with what program you work as this is not Reolve specific, your backup strategy needs to be solid, consisting of min 3 diff locations and keeping versions back to how ever long you need on each location (can differ per location based on requirements). And of course fully tested/verified regularly
Your current strategy, if we can call it that, is sorry to say not the best and a backup strategy that relies on a recycle bin (first i ever hear of such) or keeping only a backup copy(s) of the current active db and not older versions (which would have solved your issue) is not a backup strategy.
So do some reading , set it up properly and learn from this experience to prevent it from happening again.

Its always incredibly painfull to learn the hard way and i am sorry that this happend to you. Surely we all somehow faced it someway in our careers that famous 'crap' moment, but give it a positive swing and learn from it .

p.s. and this is not trying to change your mind , which you indicate you wont) but to provide you with proper info to make the right choices going forward. You are of course free to ignore and do it your way , but you just found out the effects of doing it your way.
Beatstep & APC-40 Resolve Edition Controllers https://posttools.tachyon-consulting.com
Test Rig : 2xXeon (24c) | UNRAID VM's OSX 10.x | 128GB | 5700XT | 10gbe
Prod Rig : i9-7940X (14c) | OSX 10.14 | 64GB | 2xVega 56 | 10gbe | Tb3 | V:Eizo | A:5.1RME
Offline

Jeff Ha

  • Posts: 144
  • Joined: Tue Apr 14, 2015 10:38 am

Re: Ditch the database approach to managing projects

PostWed Jul 07, 2021 10:49 am

I've looked at Seth's github for backups. It looks like it's something we can once as we migrate from Premiere. One of the other things I was wondering is how easy is it to migrate from one DB to another through Resolve? We have active projects on our SAN. Once they are completed, we move to an Archive network location for a year. Then those projects are migrated completely offline to tape.

What's the best way to achieve this with Resolve, or does our workflow need to change? Do we export individual DRPs from the Archive DB and back those up? Or is it better to say after a year, whatever is in that archive DB is archived and we start with a new Archive DB?

Theoretically, how many projects can reside in a DB (thinking archive) before it craps out?
Offline

RikshaDriver

  • Posts: 435
  • Joined: Sun Aug 12, 2018 10:08 am
  • Location: Melbourne
  • Real Name: Asim Siddiqui

Re: Ditch the database approach to managing projects

PostThu Jul 08, 2021 5:18 am

It's as easy as selecting the relevant projects in one DB and Ctrl-C

Then Ctrl-V into the new DB
Resolve Looks, Plugins & Transforms: https://xtremestuff.net/store
Open Source Resolve transforms: https://github.com/xtremestuff
Offline

Jim Simon

  • Posts: 17695
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Ditch the database approach to managing projects

PostThu Jul 08, 2021 7:13 pm

RikshaDriver wrote:It's as easy as selecting the relevant projects in one DB and Ctrl-C

Then Ctrl-V into the new DB


But it could (and should) be made even easier. ;)

viewtopic.php?f=33&t=96952

Return to DaVinci Resolve Feature Requests

Who is online

Users browsing this forum: No registered users and 7 guests