Databases went missing...Again....project lost

None of us - as far as I know - has seen the Resolve codebase. None of us know how foundational the DB is. None of us know how much dev time it would take to make Resolve able to open and save directly to drp's.

Besides that, Resolve could both leave the DB the way it is and open and save to drp files. Doesn't matter to the user if Resolve is internally using the DB. What matters is that the "normal" File/Open and File/Save commands are available for operating on projects. There are a few other pathways - like backups - to get right but that's pretty obvious as well. No need for huge code churn, higher prices, and years of instability.

Then it belongs here: viewforum.php?f=33

In this thread, we all tried to help a user who lost his work.

Then it belongs here: viewforum.php?f=33

In this thread, we all tried to help a user who lost his work.

True. And from the start he was saying how frustrating he found the system. So this thread was always about both how to get his work back and why was it so hard for him to do what he wanted to do.

It's not uncommon that help requests turn into feature requests. Though I agree that at this point some sort of concise feature request in the FR forum would be a good idea.


Sent from my iPhone using Tapatalk

True. And from the start he was saying how frustrating he found the system. So this thread was always about both how to get his work back and why was it so hard for him to do what he wanted to do.

He got his work back and realized his mistake. Then it became 79 messages about "Why doesn't Resolve work the way Premiere does?" It makes you wonder if there are people who go over to Adobe's forum and ask, "hey, how come Premiere doesn't work the way Resolve does?"

A lot of people want to blame the software and not admit, "ah -- I see what happened. I didn't understand how and where projects go." And it all boils down to a) not reading the manual, and b) not going through the tutorials. Do those two things, and I bet 75% of the questions on this forum would disappear.

This is getting Reduser vibes. Demands of shutting up criticism, "if you don't like it go away" mentality, oh man.

It makes you wonder if there are people who go over to Adobe's forum and ask, "hey, how come Premiere doesn't work the way Resolve does?"

I've never used Premiere, but having been on various audio forums, I'd say... yes, definitely! This kind of thing even happens with plugin companies where features will be recommended because some other plugin company (usually FabFilter lol) has it. This is certainly not unique to Resolve at all.

This is getting Reduser vibes. Demands of shutting up criticism, "if you don't like it go away" mentality, oh man.

Ha ha. I was think more that it had school playground vibes... at any moment the high pitched chants of 'Fight, fight, fight" could begin. :)

I wake up every morning for the past few days to see this topic updated at the top of my list of 'View your posts'. It's not overly entertaining, but I am curious to see if the original question of a database gone missing is ever answered. If it has been answered, I might have missed it in between the comments about a feature request.

It's been answered numerous times, but the OP doesn't appear willing to accept that he's doing it wrong and that protesting that it should work otherwise won't get him the correct result.

It's been answered numerous times, but the OP doesn't appear willing to accept that he's doing it wrong.

I never got a clear picture of what the OP was doing wrong. While not ideal, there is nothing wrong with having a database on an external drive and moving it from one machine to another, provided you use the 'Add Library' > 'Connect' approach the first time the database is accessed by a particular machine (and also that you make sure the external database is attached before launch Resolve and disconnected after exiting Resolve and of course, that the mounting location is the same each time - i.e., same drive letter in Windows). I couldn't tell whether the OP was simply using the incorrect terminology to describe what they were doing in this regard or whether the OP was actually using some other workflow altogether.

With all due respect, you're not reading these posts either if that's what you've come away with. There's nothing stopping him from moving a database between systems on an external drive. Probably not the best way to move projects among systems, but not difficult or impossible. But he's evidently not willing to do it the right way (the one that will actually work) or continues to object because the right way wasn't obvious to him from the start.

The procedure is simple and has been detailed more than once here.

Would it be so difficult, John, to be more specific then and explain what the OP did wrong when trying to move an external drive from one machine to another? The lack of specificity in your reply leaves me asking the same question. What part of moving an external drive from one machine to another went wrong for the OP?

His explanations are too vague and imprecise to know what he actually did, other than create lots of redundant databases each time he moves the disk. What we mostly have are arguments about why it doesn't work the way he thinks it should.

Let's assume the external disk is attached to system 1 and you want to move the disk (and the database) to system 2 for the first time. So we attach the disk to system 2 and:

1) open the database manager (aka "Project Libraries")
2) go the bottom and click "Add Project Library"
3) click "connect" and enter a name (or, preferably, "the" name)
4) browse to the location of the database folder on the external disk
5) connect

That's it. Now, when he moves the disk back and forth between systems, it's possible that one or both systems will register the database as missing even with the external drive reconnected. Resolve doesn't like databases on external drives to begin with and who knows what anyone has been doing on the computer since the disk was removed.

To forestall that eventuality I think I would probably disconnect the database before removing the drive and repeat the connect process above every time I reattached it. Whether that's strictly necessary, I don't know. You'd have to try and see. It's possible Resolve will "see" the reattached database without issue. It's also possible it won't.

Again, this is not a recommended procedure. But it should work.

Thanks for reiterating this, John. It's a good summary. We also know that putting a disk database on a drive formatted with ExFAT is destined to fail (and maybe that was also a problem for the OP). Cheers.

None of us know how much dev time it would take to make Resolve able to open and save directly to drp's.

I obviously have no specific insight into the Resolve codebase, but this type of change is very unappealing from a business perspective. You're touching stuff that is critical to successful use of the product, and is deeply embedded in the DNA of the software. It makes much more sense to adapt the existing functionality to new use cases (hence cloud libraries), rather than reinvent the wheel by altering something that can never break.

No need to touch something deep. It's just how these things are presented in the UI.
Right now you can:
Import a drp
Open the resulting project
Export the drp
Delete the imported project

This all could be wrapped in UI commands:
Open drp
Save drp

In the interests of beating this one to death, a .drp file is just a compressed project.db file, with the added capability of unwrapping itself into the current database. It has no independent life outside a database, any more than "project.db" does. It's portable, but not autonomous. For that matter, "project.db" files are also portable, if you wanted to fool directly with the database. But the .drp makes that unnecessary.

It's not as easy as you think.

And maybe it's easier than you think.

a .drp file is just a compressed project.db file, with the added capability of unwrapping itself into the current database. It has no independent life outside a database, any more than "project.db" does.
...
It's not as easy as you think.

A drp file is BMD's recommended way of transporting a project from one Library to another. It's the portable version of a project. Totally reasonable to ask for that to be Opened/Saved. Nicer workflow for those who have to move from one system to another.

If you're referring to power bins not coming along for the ride - yup! So what? Way easier for the user to manage that on a one-and-done basis than to deal with all the extra project stuff every time one wants to take one's project to another system.

Plus that stuff is far from obvious. Open/Save is totally obvious.

Added this Feature Request: viewtopic.php?f=33&t=175630

I don't think anyone is claiming that "Open/Save" is anything but "totally obvious". But since what you propose is at odds with collaborative work and the organizing principle of the software, it does present something of a dilemma -- the need for two parallel project file systems, which would also have to be interchangeable.

Whether that's difficult or easy, no idea. I would propose that the whole database/project library thing is also "totally obvious", costing the user no additional time or effort to manage, but that doesn't seem to go down well here.

I would propose that the whole database/project library thing is also "totally obvious", costing the user no additional time or effort to manage, but that doesn't seem to go down well here.

Did you read the Feature Request? It details what users who want to move their project to another machine have to go through now, and what they'd have to with the proposal. If you're saying the doubly-long list of steps is wrong please feel free to enlighten me. Right now the "no additional time or effort" doesn't appear to be true.

If you're routinely moving from machine to machine, why would you be using a local database? You've got network and cloud databases for just that purpose.

But you simply must have another filing system, whatever that may entail? How much accommodation do they owe you?

People complained bitterly about lack of external monitor support without Decklink, and they delivered Clean Feed. So maybe all this ends with autonomous project files. Good luck.

John - I don't mean this to be hurtful to anyone. I'm sorry this rankles.

I personally don't need this feature. I'm fine toting Resolve DBs around on external drives. But I want Resolve to succeed in the larger prosumer/indie market and I think this one is a big deal. Hence my suggestion.

I'm not suggesting another filesystem. I'm suggesting the filesystem that every computer has - and every user is familiar with. Resolve has its own filesystem - the db - and I'm not suggesting tampering with that. Just asking for a simple interface to something that's already understood by everyone and already supported - somewhat indirectly - by Resolve's .drp's.

[Peter Chamberlain]

Resolve has 'Import and export project' in the file menu, because you are importing it to a project library.

There isn't an open operation as projects need to be in the project library.

note, we allow users to drag a drp into an empty timeline or project manager, to import it.

my recommendation is to not put project libraries on portable drives, particularly if your OS can arbitrarily change the drive name/letter.

Totally understood Peter. If you look at my Feature Request I list what I believe are the necessary steps for a workflow with .drp's - and it's both long and complex. The feature request is designed to make it simpler.

Feature Request: viewtopic.php?f=33&t=175630

If there's a better workflow that I'm not seeing I'd love to hear about it!

[Shrinivas Ramani]

For workflows where there's a lot of interchange between machines, we've traditionally recommended shared project libraries - be it self hosted or on the cloud.

For off-the grid workflows, you can even choose to locate self hosted project libraries on the more portable machine, save locally when on the move, and connect through the network when you wish to switch. With the Blackmagic cloud, it's the same, multiple machines, you don't need to assume hosts, you don't need specific machines to be online to connect to the library. In both cases, with live save, project changes are available instantly, and you can even switch machines mid-edit to take advantage of renders or different hardware/OS capabilities.

The DaVinci Resolve installer includes the option to install compatible PostgreSQL versions if needed. So switching to self-hosted libraries requires three steps to try - switch tabs in the project library, create a new network library and copy/paste the required projects over.

Of course it's your decision. I just think the more you grow the stronger the pushback is going to get.

Thanks for commenting!

I just did a little experiment with the disk database. I'm on Windows.

1) I created a blank project named "Test DB".
2) I closed Resolve and opened the location of the disk database in the Windows file explorer.
3) I took the project.db file from an existing project and copied it over the project.db in the "Test DB" project location.
4) I opened Resolve and opened the "Test DB" project.

The "Test DB" project was now an exact copy of the other project I took the project.db file from.

Conclusion: An external program or script could be created that could copy the project.db file to a portable drive, and copy the file from the portable drive into a project in the disk database on a different computer. You could do this any time you wanted to move the project to another computer and back again without exporting, importing, renaming, deleting, etc. Just run the program, select which project to copy, select a destination (the portable drive), and copy. On the other computer, run the program, select the project on the portable drive, select the project (or create a new project) in the disk database, and copy.

The current disk database in use is stored in a file named "dblist.conf" in "C:\Users\Username\AppData\Roaming\Blackmagic Design\DaVinci Resolve\Preferences\" so it shouldn't be too hard to traverse the disk database to retrieve the list of current projects.

[Shrinivas Ramani]

Gary

Just re-emphasising this bit from my response above.

where and how a project is stored - be it a file.db or a specific folder structure or an entry in a database table or a wisp of string in the cloud - is meant to be opaque. Between versions, this structure can and will change to accommodate newer features. The intended interaction is via Resolve or the Resolve project server application - which allows import/export (for project exports), archive/restore (for project archive folders), backup/restore (for project libraries) and even copy-pastes between folders and libraries.

Resolve doesn't stop you from copying individual project.db or other bits and bobs across database folders - or using SQL queries to change info. It just means you agree to carry the risk of potential data loss.

Regards
Shrinivas

Shrinivas, understood. If I were to create such a program, emblazoned at the top would be "USE AT YOUR OWN RISK".

The program could check which version of Resolve is installed on your computer (It's in the registry) and save that info along side the project.db file. On the other computer, the program would compare the version installed on that computer with the info on the portable drive, and if different, not allow the procedure.

If enough people are interested in a program like this, I may attempt it.

To me the key to this is that it be obvious in the UI. If one could add a menu item in the right place in the File menu one could probably write this safely via scripting. But then it would be an addon which would pretty much defeat the purpose as people would have to know enough to add it.

It's really just up to BMD. If they don't want it then it won't exist.


Sent from my iPhone using Tapatalk

I just did a little experiment with the disk database. I'm on Windows.

1) I created a blank project named "Test DB".
2) I closed Resolve and opened the location of the disk database in the Windows file explorer.
3) I took the project.db file from an existing project and copied it over the project.db in the "Test DB" project location.
4) I opened Resolve and opened the "Test DB" project.

I basically do this myself when updating Resolve that requires a database update. I recall reading on the forum a while back (and I believe it was from someone at BMD, but don't quote me on that) that creating a new database is better than continuing to use an "updated" database from a previous version. I'm not sure which thread it was, otherwise I'd link to it.

This is my process:
1) Export the database and export individual projects (just to be safe)
2) Update the database to be compatible with latest version
3) Create a brand new database
4) Move my updated files into the new database via Windows Explorer
5) Verify the new database works
6) Delete the old database that was updated

That has worked just fine. But I imagine BMD wouldn't recommend it though. lol

It just means you agree to carry the risk of potential data loss.

How is this an acceptable status quo?

If BMD wants that army of content creators transition to Resolve, make it more entry and user friendly. One of Apple's recipes for success was from early on to shield creatives from technical necessities. Resolve's database structure is nothing but a technical necessity. You act like it does not need to be improved, when it can and should be vastly improved. There have been suggestions.

I hate to say it but BM should looka t FCPX The Progject Libraries are self contained. All the clips, audio etc are in one place along with the project database files. Thus they can be run from an external hard drive and moved from computer to computer with no problems.

FCPX and Resolve have come from very different histories and directions of travel. However the majority of Resolve's user base is, I would guess, not wnat it was 10 years ago but now much closer to the one FCPX has. People who do Lots of small videos, often several simultaniously. Also loading to multiple youtube/vimeo and other platforms. They will work from external hard disks they can pass around.

So BM need to square the circle without dropping the ball. (or mixing too many metaphors)

I have seen a similar problem with software development and compiler systems. The answer (possibley) is that you can have a self contained Project folder that hoilds the clips and the project database. The only problem I can see is that a project database, currently, has to be created in an empty folder. I can't set up a folder, dump a load of clips and other media folders in it and then create a database in the root folder.

Also the open/close projectslibraries is not as simple or as intuitive it could be. Certainly not for new users and the majority of Resolve users are fairly new.

So the project uses the project paths etc (all relitive to the project root) but other settings and paths will be local to the copy of Resolve. It is not difficult and how we managed it on large software projects. (400 people on 10 sites across 6 countries) Though BM have to do it without breaking 1,000s of current projects ::-)

It's easy to explain what is wrong fromthe comfort of an arm chair. Fixing it in the trenches is a different matter. Good luck to them.

Though BM have to do it without breaking 1,000s of current projects ::-)

yep. please.

Here's a video of what I think the .drp workflow looks like at present - that is if someone wants to pass .drp files around to either share an edit or edit on multiple machines without setting up a networked solution:

Boy i wanted to move on from this. I had gotten my databases restored and had gone back to work. Then i moved my external HD to the desktop and found that the exact same project was an older version. I mean how is that even possible?

Same drive, same media, same database, same project filename. Older version of project on desktop, current version on laptop. 2 different versions of the exact same project. Everything on the hard drive.

Per the community I exported a DRP file off the laptop and imported the DRP into the desktop machine. So I got an updated version. But now I have an older and current project version on the desktop and a current version on the lap top?? All within the same database.

Am I the only one that sees the media management mess this creates?

I cant even begin to fathom how this works, how can you get an older version of a project on your desktop mac and the current version on the laptop. wow....Ill jsut reiterate that the deeper i get into this the more apparent it becomes that Resolve is jus tnot built to be a mobile NLE.

Lots of great ideas on how to make this work better so appreciate everyone thoughts. Hopefully BM will address soon in a future update. I just cant see how this system works better then any other NLE.

Besides that, Resolve could both leave the DB the way it is and open and save to drp files. Doesn't matter to the user if Resolve is internally using the DB. What matters is that the "normal" File/Open and File/Save commands are available for operating on projects. There are a few other pathways - like backups - to get right but that's pretty obvious as well. No need for huge code churn, higher prices, and years of instability.

this is the best idea ive heard yet.... thanks Joe! Make resolve normal!!

@Shrinivas Appreciate you jumping into the conversation but after doing some research it looks like shared project libraries are server based. And thats not going to solve the issue here.

Maybe theres a workflow here im missing but I think its something BM has to explain in greater detail. Preferably on the youtube channel. Maybe the cloud based databases are a better solution......

but can anyone explain how i have the same databases, media files, project names etc, but the project is in a completely different state depending on which machine I plug into?

Here's a video of what I think the .drp workflow looks like at present - that is if someone wants to pass .drp files around to either share an edit or edit on multiple machines without setting up a networked solution:

Joe this is a great idea, thank you for sharing. If you don't mind one point of feedback. Instead of replacing the DRP file, one might want to save it with a current date and time in the file name. Or even just a V1, V2, V3 etc....

This would be similar to the autosave folder in premier pro.

But you hit the nail on the head that all these workflows wind up making copies of copies of copies. And media management becomes dicey.

Anyway thanks again for sharing that

It's been answered numerous times, but the OP doesn't appear willing to accept that he's doing it wrong.

I never got a clear picture of what the OP was doing wrong. While not ideal, there is nothing wrong with having a database on an external drive and moving it from one machine to another, provided you use the 'Add Library' > 'Connect' approach the first time the database is accessed by a particular machine (and also that you make sure the external database is attached before launch Resolve and disconnected after exiting Resolve and of course, that the mounting location is the same each time - i.e., same drive letter in Windows). I couldn't tell whether the OP was simply using the incorrect terminology to describe what they were doing in this regard or whether the OP was actually using some other workflow altogether.

No steve, despite what jon wants to fantasize, it was never established why the databases went missing. I was able to restore them and continue to work but why they got disconnected is still a mystery. Perhaps if it happens again, the community has provided enough added insight into databases I might be able to troubleshoot better and figure out what is going on. The best idea I have right nowis that it might have something to do with an upgrade from 17 to resolve 18. But that doesn't explain why 2 databases out of 8 disconnected but the other 6 were fine.

I created a new post in the feature request forum for us to argue about databases over there. Hopefully it will be more visible to the BM team. My final point ( I hope) is that i love resolve and its such a great NLE that this conversation leads to a better product. Its never been about Resolve vs. premiere vs fcp. But those are solid NLE's too. If they do something well that resolve doesnt, then lets let the BM team know about it without getting all bent at the feedback.

viewtopic.php?f=33&t=175904

If anyone wants to jump in on the dated project issue i ran into tonight feel free. Im guessing there is a local os directory tree that tells the database where to go, based on which ever machine the drive is plugged into.