Project Size Limit?

Can a project be too big? I have a been building on the same project for about 6 months - it's almost finished, but instead of making a new project for each episode, we've just been adding episodes into the same project. Now, about 50% of the time DR (14.3) just does the 'vaporize' crash on launching the project. We've backed it up multiple ways, but I'm wondering if there's a limit to project size? We haven't added much media to the project from the very beginning - we attached all source media for the entire season right at the beginning of the project. (Online conform and color correct only, not offlining project in DR.)

Once the project launches, DR works great - no issues. Any ideas?

12 core mac cheesegrater 5,1,
mac OSx 10.12.6, DR 14.3
Nvidia 680 (gui) nvidia Titan XP (renders)
project on internal raid, source on attached SAS raid, simple graphics and some media from shared storage (facilis)

Could these be some timeouts when reading from shared storage?

[Peter Chamberlain]

Hi, disk or PostGres database?
If you export the DB how big is the export?

Hi, disk or PostGres database?
If you export the DB how big is the export?

Disk db, I think the last backup was around 600 Mb. I will run another tomorrow.

Could these be some timeouts when reading from shared storage?

The DB is local- only a few graphics are on the shared storage, and that sorage has been working flawlessly otherwaise... but it’s possible. time-outs certainly shouldn’t cause the app to disapppear.

[Rohit Gupta]

Hi Greg,

Are you able to post a crash log? Also, if you could PM me a link to the project export, that might be useful in analysing the crash report.

Thanks,
Rohit

Can a project be too big? I have a been building on the same project for about 6 months - it's almost finished, but instead of making a new project for each episode, we've just been adding episodes into the same project.

Man, that is very dicey. What I've always done is created one folder per project & season ("Big Network Show - 2018"), then store files based on episode number ("S01-E09"). Every episode for that season would still have access to stills for previous shows with the Gallery manager. Making gigantic projects yields to crashes, memory problems, and massive save/export times. I think this is asking for trouble.

Even with half-hour sitcoms, I've never put more than one show in one project. But I have put fixes and promos for shows within that project, just to try to keep everything together.

What I would do in your case is to export multiple copies of the project file and just eliminate the stuff you don't need. Maybe you could do Episodes 1-3, 3-6, 7-9, and so on in one project file. Just delete what you can eliminate in order to make the session more lean & mean.

Hi, disk or PostGres database?
If you export the DB how big is the export?

Thanks for your patience and for checking back.

the project drp is about 570 Mb; the db is about 1.4 Gb

Even with half-hour sitcoms, I've never put more than one show in one project. .

This makes some sense, but the amount of source material doesn't really change - just the data of the timelines. I never intended to organize the project like this, it just turned out that way. Because I have a large source pool, that crosses episodes, I had intended to media manage each episode separate - but found that unreliable (at best!) and pretty much ALL my sources are in the media pool, which is common to all episodes. I normally do a separate project per episodes.

I may try your 'deconstruction' approach to see if it works. The source media's not organized per episode, so I don't know how much eliminating timelines will reduce the size of the individual project. Maybe the aaf and grading info is much more data than I realize.

Hi Greg,

Are you able to post a crash log? Also, if you could PM me a link to the project export, that might be useful in analysing the crash report.

Thanks,
Rohit

Thanks Rohit! Naturally, since you requested that info, the app hasn't crashed opening the project. (we used to call that engineering proximity effect.) When/if it crashes, I'll grab the crash log and project and send them to you.

I may try your 'deconstruction' approach to see if it works. The source media's not organized per episode, so I don't know how much eliminating timelines will reduce the size of the individual project. Maybe the aaf and grading info is much more data than I realize.

I had to do this a year or so ago when 14 came out and the database structure radically changed (at least from my perspective). I had something north of 450 projects in a single database, and I realized it was way, way too cluttered and confusing. I finally opted to do one database per year, and that chopped them down to maybe 80-100 projects per year -- some of which were folders. If I had a series, that would just be one folder inside the yearly database. And I could see a very complex facility (or a very busy colorist) even going to new databases per client.

The real key to me is the save time and the backup time, and if you have a single project that has dozens or hundreds of timelines, it can really muck up and slow down quickly. A project with (say) 5 timelines is going to save and backup a lot faster than one with 50 timelines. It's not just the size of the project per se -- it's how Resolve responds in terms of speed.

In terms of organization, I would have the exact same approach if this were for Baselight or Nucoda or any similar program. Same thing with editing.

We do ones database per client/project, whatever Make sense:
-a movie
-a season of a episodic, each year new database
-a commercial client. If it is big, different database for year

Inside the database, a project per "session"
- each reel of a movie
- each trailer advertisement campaign
- each ine of the episodes of the season
- each commercial from that client.

That ensure lean projects, easy recover from failure, easy collaboration.

We do ones database per client/project, whatever Make sense:
-a movie
-a season of a episodic, each year new database
-a commercial client. If it is big, different database for year

Inside the database, a project per "session"
- each reel of a movie
- each trailer advertisement campaign
- each ine of the episodes of the season
- each commercial from that client.

That ensure lean projects, easy recover from failure, easy collaboration.

Walter,

Do you have a system in place to keep track of all the database names? Do you keep every database you are working on loaded in you home page? Curious how you keep that many databases straight.

Sent from my Pixel 2 using Tapatalk

We do ones database per client/project, whatever Make sense:
-a movie
-a season of a episodic, each year new database
-a commercial client. If it is big, different database for year

Inside the database, a project per "session"
- each reel of a movie
- each trailer advertisement campaign
- each ine of the episodes of the season
- each commercial from that client.

That ensure lean projects, easy recover from failure, easy collaboration.

Walter,

Do you have a system in place to keep track of all the database names? Do you keep every database you are working on loaded in you home page? Curious how you keep that many databases straight.

Sent from my Pixel 2 using Tapatalk

mostly is named with the project name plus time, and when a database is "dormient" you simply archive it and disconnect it.

simple and clean.

For Resolve-based facilities, I think it also helps to have a shared PostgreSQL database between rooms. That way, everybody can see the filenames of everybody else's project.

I would think for a facility, it'd be a good idea to have a scheduling program and an operations manager who keep track of project names and naming conventions. Generally, the longterm projects I've done all start with a meeting and a group email that specifies how every project needs to be named and how each episode is noted. I have seen facilities that have adopted Netflix' way of doing things, and that seems to help for longterm projects with multiple workers and even facilities. Get it in writing so there is no doubt, particularly on formatting, slates, and so on. (I keenly recall one time being chastised by a Warner Bros. exec because I actually spelled out "Brothers" on a slate.)

I have generally gone with all lower-case and underscores for spaces in project names (and no high-ASCII) since that's also how the Resolve databases work (at least the XML portion). So it could be something like superhero_s02_e09, and then specific versions for each pass and a final for the delivered version. Deliverable file names go all over the map, and for that we just use whatever the client wants. I generally insist that we embed the date of the project in there somewhere, just to avoid confusion.

I have a been building on the same project for about 6 months...Now, about 50% of the time DR (14.3) just does the 'vaporize' crash on launching the project...

We attached all source media for the entire season right at the beginning of the project. (Online conform and color correct only, not offlining project in DR.)

Once the project launches, DR works great - no issues. Any ideas?

..Can a project be too big?

12 core mac cheesegrater 5,1,
mac OSx 10.12.6, DR 14.3
Nvidia 680 (gui) nvidia Titan XP (renders)
project on internal raid, source on attached SAS raid, simple graphics and some media from shared storage (facilis)

Here's one idea, til you hear from BMD or a more knowledgeable user with an answer:
It could be that your project file is become corrupt.

If you have not, be sure to go to Resolve's Project Manager then right-click on the project, choosing the
"Export with Stills and LUTS" option, for a more comprehensive .drp (project) backup to the folder of your choosing.

While you're here, Greg, kindly inform us
• What kind of video files (i.e., the CODEC type) did you start with, and
• What CODEC type did you transcode them to, and
• At what data-rate are the transcoded files?

RS 14.3.0.014 | MiniMonitor | DTV 10.9.7 | Win64 8.1 | ASUS ROG G751JL, Intel i7HQ, 24GB, 1TB HDD, 500GB EVO 850 SSD, GTX965M | Mackie MCU Pro | Softube Console 1 | Shuttle Pro 2

While you're here, Greg, kindly inform us
• What kind of video files (i.e., the CODEC type) did you start with, and
• What CODEC type did you transcode them to, and
• At what data-rate are the transcoded files?

When the project has opened, I've been able to make a .drp, which then imported into a new database, but the duplicate project exhibits the same opening difficulty. Mostly crashes the app while loading - but every so often it can be opened. I have not tried to open an older backup, as I don't want to loose weeks of grading work.

The project worked fine for months, and only developed this odd behavior a few weeks ago. I'm thinking it's something added about that time that created this issue. The only things we've been adding periodically are graphics (.psd) and AAFs. No new footage.

Today it took 3 tries to open the project, not bad. Yesterday I worked on it most of the day after 3 or 4 attempts to open.

While I don't entirely grasp the relevance of Peter's questions to a crash that happens when opening the file, I'm at a loss to solve the problem, so I'm happy to provide that info it leads to a solution.

- The image files are almost all DNG Raw, about 65Tb worth (again, no problem until recently, and those have been attached for months. Further, we recently created a new project in a new database using almost all the same source material, but different timelines, and it has no issues.) There are also some psd files and a few pieces of stock footage - mostly prores. There are also .wav files of the mix stems.

- There are no trancoded materials kept in, or re-entered into, the project. Target renders have been both various DNxHD and ProResHQ, mostly as tests, but the software has no problem making those. We're waiting on a few small things before making final masters, so the project is 'live' until then. (Though, I'm beginning to feel the need to render out sequences as 'submasters,' just in case!)

- The data rate for DNx and ProResHQ is roughly the same for HD 23.98 - about 175.

The project is local, not shared, the media is mostly local on very fast raids, with some of the graphics on equally fast network storage (that is technically shared storage, but no other bay is working on this project.)

Hi Greg,

Are you able to post a crash log? Also, if you could PM me a link to the project export, that might be useful in analysing the crash report.

Thanks,
Rohit

Hi Rohit

I'm still experiencing this problem, though I'm eventually able to get the project to open. Once open, it acts normally.

Unfortunately, there's no crash log posted from this crash - it's one of those 'vaporize' crashes, where the app just disappears. I wlll attempt to PM you a copy of the project shortly.

While you're here, Greg, kindly inform us
• What kind of video files (i.e., the CODEC type) did you start with, and
• What CODEC type did you transcode them to, and
• At what data-rate are the transcoded files?

When the project has opened, I've been able to make a .drp, which then imported into a new database...

...The project worked fine for months, and only developed this odd behavior a few weeks ago. I'm thinking it's something added about that time that created this issue. The only things we've been adding periodically are graphics (.psd) and AAFs. No new footage.

Today it took 3 tries to open the project, not bad. Yesterday I worked on it most of the day after 3 or 4 attempts to open.

...I don't entirely grasp the relevance of Peter's questions to a crash that happens when opening the file...

...The image files are almost all DNG Raw, about 65Tb worth (again, no problem until recently, and those have been attached for months....


...I'm beginning to feel the need to render out sequences as 'submasters,' just in case!)

...The data rate for DNx and ProResHQ is roughly the same for HD 23.98 - about 175.

The project is local, not shared...

1) Ok, by "175" near the end of the above quote, do you mean to convey...
"The data-rate for your DNx... and ProResHQ [transcoded] files is...about
175 Megabits per second...MegaBytes per sec. or what, if you don't mind telling? (It's not clear what 175 means, without the useful qualifier after it.)

2) Conveying the type of files you're editing along with their and their data-rate provides some useful insight for those able to assist you -- especially since you have yet to attach a Resolve diagnostics log.

I was recently made aware by Dmitriy at BMD Customer Support that lack of integrity of a huge project that's been lying on a system for a good number of months, with increasingly more executed on it occasionally, especially an edit project with highly-compressed files -- such as I have here, in a 7-month long-in-the-tooth project, containing over 700 event elements in the timeline, most of which are an H.264 in an MPEG4 wrapper -- and others of which are 29.98 fps AVCHD at 50MB/sec.

I have more reason to be nervous I think, than even you do -- due to the stress imposed on the CPU here!

This project here, is an experimental test of my resolve to try BMD's editor for the very first time -- and on a moderately powered ASUS gaming PC, and without transcoding.

If I were you, I''d make your backups of the database as well as the project itself (with stills and LUTS if applicable) apart from the RAID too.

Here's hoping someone can guide you in looking for yet another[i/i] type crash log that may well be lurking on your machine, that might even reveal something that's gone amiss within your computer's OS environment, and not Resolve, as you have imagined.

As an example, after Resolve failed to SAVE or SAVE AS or even open, I found a somewhat useful, though not as comprehensive "Davinci Resolve Diagnostics Log" on my PC, within
C:/AppData/Programs/Blackmagic Design/Resolve/Support if I'm not mistaken about that path [i](I'm currently away from the machine).

That aforementioned "Diagnostics Log" fortunately, in plain language, revealed even to this novice crash log reader, that I had developed a WindowsMM [MultiMedia Player?] output device error, causing a conflict with Resolve-connected USB MIDI surface controllers (Softube Console 1 hardware/software audio plug-in -- and Mackie MCU Pro Universal Controller).

That WindowsMM device error, fighting within the OS environment, was preventing Resolve from opening most of the time, but opening sometimes -- and saving sometimes, but no longer able to effect a "Save As" operation.

I fixed it myself by disabling, then immediately re-enabling 3 or 4 drivers, one at a time, within Windows 8.1
Device Manager > Sound, video and game controllers

Hope all this provides you impetus for more ideas, and apart from consideratiob of insights offered you here, your retrieving and providing a useful, pertinent log is well critical in speedily having people aid you in tracking down the culprit.

By way of encouragement, let me presumptuously assert that you likely *do* have a useful log, but you may need to fish a bit more earnestly for it.

Cheers.
Peter

RS 14.3.0.014 | MiniMonitor | DTV 10.9.7 | Win64 8.1 | ASUS ROG G751JL, Intel i74720HQ, 24GB DDR3, 1TB 5200RPM[only!], 500GB EVO 850 SSD, GTX965M | Mackie MCU Pro | Softube Console 1 | Shuttle Pro 2

I'll look around for a similar crash file as the one you detail - I'm on a mac. The project tends to crash the app on exit now, too - although 'autosave' has been working fine, so I don't really loose anything.

Yes, the data rate of the rendered files is around 175Mbps, but, again, only a few graphics are traditional 'codecs,' 98% of the project is uncoverted RAW DNG 4:1, which even at 4k actually sports a lower data rate, reportedly 135Mbps.

Earlier in the thread there was the hint that something was timing out during the loading of the project. I'm starting to think there's validity in that notion. Next time I have the project open, I'll sift through the bins for anything that's offline and see if that makes a difference, plus delete any extra timelines.

I'll be away from this project for a couple weeks unfortunately, so it may be a while before I can do any more diagnosing. If I can get the project backup cleanly again, I'll start to extract the timelines in bulk, removing any 'unavailable' media, etc. And, of course, look for that crash file yo mentioned. (If I read correctly, I can also simply duplicate the project into another disk database at finder level. TBD.)

Thanks for the tips.

Sure, glad to try to be of help.
One more thing:
If at all possible, physically disconnect every single computer peripheral/accessory device not necessary to run your Resolve project.

That approach in fact, resulted in a successful series of Resolve Startups (as I had unplugged a couple of USB devices).

This system simplification and concurrent process of elimination trying to add one more perioheral at a time, aided me in ultimately identifying that it was a MIDI related conflict within "Windblows" OS preventing the Resolve Start-ups, saves, etc., and eventually gave me the intuitive audacity to search for that surprisingly telling, Resolve Diagnostics Log tucked away in the PC's AppData > Programs > Blackmagic Design > Resolve > Support (or closely similar) hierarchical structure.

IMPORTANT CONSIDERATION
In your Mac, I'd look within the Library (or is it "Libraries"?) > ...Applications > Blackmagic Design > Resolve > Support? hierarchy, or some such path -- then further encouragement likely awaits.

'Til you get back to this conundrum, be well.
Peter [quote="Greg Huson"]I'll look around for a similar crash file as the one you detail - I'm on a mac.

I'll be away from this project for a couple weeks unfortunately, so it may be a while before I can do any more diagnosing.

RS 14.3.0.014 | MiniMonitor | DTV 10.9.7 | Win64 8.1 | ASUS ROG G751JL, Intel i7HQ, 24GB, 1TB HDD, 500GB EVO 850 SSD, GTX965M | Mackie MCU Pro | Softube Console 1 | Shuttle Pro 2

I'll be away from this project for a couple weeks unfortunately, so it may be a while before I can do any more diagnosing. If I can get the project backup cleanly again, I'll start to extract the timelines in bulk, removing any 'unavailable' media, etc.

Yes, I have found you can sometimes speed up "save" operations and reduce the size of projects by deleting source files that are not actually being used in a timeline. The trick is, if you have many, many TB of data floating around a single project, it might be in one timeline but not another. I think you're on thin ice with project files with this many timelines and massive amounts of data.

If you had one project with 100 timelines, it wouldn't be that hard to just save 10 copies of the file, then go into each one and delete the 90 timelines not being used, carving each project down to (say) 10 timelines. And once you do that, Rename the session file so that it reflects what episode/timelines are actually in there, and then you can delete the unneeded source files.

I will personally eat the disk if this doesn't save you some time and cut down on crashes.

[Peter Chamberlain]

Hi Greg,

Are you able to post a crash log? Also, if you could PM me a link to the project export, that might be useful in analysing the crash report.

Thanks,
Rohit

Hi Rohit

I'm still experiencing this problem, though I'm eventually able to get the project to open. Once open, it acts normally.

Unfortunately, there's no crash log posted from this crash - it's one of those 'vaporize' crashes, where the app just disappears. I wlll attempt to PM you a copy of the project shortly.

Greg, there is always a Resolve log. They are the only logs we look at.
Please refer to
viewtopic.php?f=21&t=69837

Post a link to the log, regardless of it it’s crashed, stalled.

Marc - it’s only 20 timelines- but 5 of them are duplicates in some form or another. I know for a fact that there are also duplicate / unavailable media entries- I had previously ignored those. (Files that we’re moved and re-entered, rather than pointing the old entries to the new location of the media.). Maybe that’s an issue?

Thanks
Peter, I’ll pull that file Sunday.

Marc - it’s only 20 timelines- but 5 of them are duplicates in some form or another. I know for a fact that there are also duplicate / unavailable media entries- I had previously ignored those. (Files that we’re moved and re-entered, rather than pointing the old entries to the new location of the media.). Maybe that’s an issue?

I have found in general you get "snappier" response from Resolve if it's not bogged down with tons and tons of unused material in the timeline. The internal database still has to keep track of those files, so it's got to be doing something. If not all these timelines are being used, then I'd say delete the unused duplicates, then check the USED column in the Media Page, update it, and delete everything that's not actually in a timeline. See if that doesn't speed up response, reduce the size of the project, and reduce the number of crashes.

...I'd say delete the unused duplicates, then check the USED column in the Media Page, update it, and delete everything that's not actually in a timeline. See if that doesn't speed up response, reduce the size of the project, and reduce the number of crashes.

Great advice, Marc. Think I'll also give that a whirl for this project here.

RS 14.3.0.014 | MiniMonitor | DTV 10.9.7 | Win64 8.1 | ASUS ROG G751JL, Intel i7HQ, 24GB, 1TB HDD, 500GB EVO 850 SSD, GTX965M | Mackie MCU Pro | Softube Console 1 | Shuttle Pro 2

...

1) Ok, by "175" near the end of the above quote, do you mean to convey...
"The data-rate for your DNx... and ProResHQ [transcoded] files is...about
175 Megabits per second...MegaBytes per sec. or what, if you don't mind telling? (It's not clear what 175 means, without the useful qualifier after it.)

...

Huh, it's clearly 175 megabits per second. There is even a version of DNxHD called "DNxHD 175" which a constant bit rate 175 mbps codes. 175 megabytes would be huge for HD. I believe even uncompressed HD 4:2:2 is less than 175 MBps.

Yes, DNxHD 175 is actually 175 Mbps (megabits per second):

https://en.wikipedia.org/wiki/List_of_A ... esolutions

DNxHD is roughly equivalent to the bitrate of ProRes 422 (147 Mbps), but 422HQ is at 220 Mbps.

And one difference between DNxHD and DNxHR and Prores is that DNx is constant bitrate codec. So 175 is always 175, that's not the case with Prores. Prores will increase or decrease bandwidth according to the complexity of the image being encoded. This gives Prores a minor space usage advantage, as it doesn't "waste bits" when encoding something like a black frame.

But what DNx's constant bit rate does allow is for replace edits on completed files. Similar to what can be done with SR tape. So if, for example, there is a typo in a lower thirds you can do a frame replacement without having to do a rerender of the entire file.

Why Avid never built this ability into Media Composer, IDK. But that was one of the design goals of DNx and it can be accomplished with third party apps, and perhaps ffmpeg.