Media Management 20+ hours to export

I'm exporting an indie feature for the colorist using the Media Management tool (my first time). I selected used media with trim 24 fr, it says there are 388 clips, old size was 580GB, new size 412GB and showing the progress as a 20+ hours export job! Is this normal? It seems excessive to me. Why would this take so long?

Also does anyone know if I click on stop can I resume later and not lose the several hours of what it has already done?

Well the answer to that is no. Clicking stop, stops the process and you can't resume which is a shame

Is this normal?
- depends on disk speed mainly, low to zero GPU & CPU use if you don't transcode, but if your target drive is a cheap portable SMR drive then it will slow to dial up modem speed after it's cache is filled

if I click on stop can I resume later and not lose the several hours of what it has already done?
- stop it and you'll need to start over

and a shout out for "Resolve Collect"- will help with a zillion weak links in the native tool, ie: 19 hours into a transfer Resolve finds a frame it does not like, and quits, only option is to delete all the media and start over ... very frustrating, and i'm being polite, but can't help with a SMR drive

I'm exporting an indie feature for the colorist using the Media Management tool (my first time). I selected used media with trim 24 fr, it says there are 388 clips, old size was 580GB, new size 412GB and showing the progress as a 20+ hours export job! Is this normal? It seems excessive to me. Why would this take so long?

I'm guessing it's very large files. Media Management is limited by the speed of your system and the speed of the connected drives. I just had a 20TB (source) conform that took about 20-22 hours to slice all that footage down to 3TB of 5K R3D originals, and I was fine with it. The client had given us relatively-slow USB3 drives, and the new conformed source media-managed drive was an SSD RAID0. We accept that it takes this long, especially when it's accessing material spread across multiple drives.

Also does anyone know if I click on stop can I resume later and not lose the several hours of what it has already done?

It will not. A bigger problem is, if Media Management encounters an error, it will come to a screeching halt and refuse to go on. I wish it had better error trapping, so it would just finish the entire Media Management process and then pop up a warning that says, "Caution! The following errors were encountered!" and then list whatever files it couldn't find. Dermot above is right in that Resolve Collect will not do this, but Resolve Collect also will not trim footage, so there's that limitation.

Here's some Media Management tips that work for the way we use Resolve:

1) limit your session to just the files actually used in the session (that is, make sure no unnecessary files are sitting in bins)

2) Render-in-Place all H.264, JPG, TIFF, and PNG graphics clips to ProRes or DNxHR so that now the clips have embedded timecode and (preferably) unique file names

3) for camera clips with embedded audio, my opinion is you're better off if you strip the sound out as a WAV file that lives in the session

4) be aware that Titles can be a bit dodgy and don't always survive the changeover with Media Management. (I would say the same thing with Fusion sequences, which I would render out and treat as a separate transcoded element.)

The simpler you make your session, the better the potential for successful Media Management. The moment you have a filename clash or a timecode conflict, it can fail. I wish Resolve had better error trapping so that when it did encounter an error, it just popped up a message with a list of problems, rather than just bailing on the Media Management entirely.

As mentioned earlier, if your file copying is failing for another reason, try Nikolai Waldman's Resolve Collect and I bet it'll get you at least 98% there without errors. His program has been a lifesaver for me over the last 6-7 years.

http://www.niwa.nu/resolve-collect/

Another possibility (which I haven't tried) is EditSpy for Windows:

http://edlspy.felixhuesken.de/

Thank you for the tips!

I had already trimmed down the timeline as per the colourist requirements. I deleted all the audio.

I started it early evening last night but it failed. Media Manager is saying new size as 411GB but it copied 1.36TB and that was all the space on that drive. I guess that figure isn't accurate. I'm trying again on a different drive that has 3TB hopefully that will be enough. I wish it would actually be able to tell me how much space I need.

Media Management can’t cut down all source formats or codecs. Which ones do you have?
Give Resolve Collect a try, as Marc suggested. It’s better.

Media Management can’t cut down all source formats or codecs. Which ones do you have?
Give Resolve Collect a try, as Marc suggested. It’s better.

I'm giving MM another try but really the inability to troubleshoot or resume makes it almost unusable.
I have mostly RED footage with a dozen of mp4 stock footage. If that fails again I'll try Resolve Collect.

Red wavelet or DCT? The former should work, not sure about the newer format.

I'm at 44% with MM and it's using almost 3TB I will run out of space again.

Does Recollect give you an accurate amount of space needed for the data?

I'm at 44% with MM and it's using almost 3TB I will run out of space again.
Does Recollect give you an accurate amount of space needed for the data?

So I installed the demo and I wasn't seeing anything size estimate until I clicked copy files. It tells me I need 1.6TB and it would be done in less than 4 hours. That's fantastic but why is the DaVinci Media Management so off the mark, needing more than 20 hours and copy 6TB of data?

I'm giving MM another try but really the inability to troubleshoot or resume makes it almost unusable. I have mostly RED footage with a dozen of mp4 stock footage. If that fails again I'll try Resolve Collect.

MP4 is just a container. Assuming you've got H.264 clips in a timeline, my advice (from above) would be to render-in-place as DNxHD or ProRes, and THEN you have a better chance of Media Management working. Without unique filenames and embedded timecode (which often H.264 does not have), it can lead to disaster in post workflow.

While you are right (of course) about potential problems with mp4, I don't think it makes much sense in this context. If the TO want's to save space when archiving and has lots of mp4 footage, it might be better to simply archive instead of media manage with trim. It will need careful checking for correctly linked footage nevertheless.

Generally, I suggest to check mp4 for valid TC before even editing it. DR is actually pretty good at finding it, FCP-X has more difficulties there. If it isn't read, i.e. all clips starting at zero, one should addd it with a tool like Edit Ready. When that has been taken care of, Media Management should work too.

I'm still suspicious about the Red footage being the issue of the TO.

So I exported without a hitch (using ResolveCollect) and sent it to the colourist only to find out none of the multi-cam got copied. Is that a known issue with Recollect? Anyone has any idea of how I can get Recollect to copy multicam data?

Apparently you have to flatten multi-cam clips or they won't get copied. I just wasted one day of colouring because.

I thought I'd report back on what I learned from using Media Management and ResolveCollect should it benefit anyone in the future. 

A little recap and some background.

I fully switched to DR last year and this is my first feature.
The project was 78 min Indie feature shot on RED. I had about 18TB of raw footage scattered over 4 drives.
 
Tiny budget but they had managed to get a pro colourist. I had prep my project according to his specs. I made a copy and removed all unused footage from the media pool. I ran Media Management, (with settings as follow: choosing timeline, copy, used media and trim, relink to new files) It was telling me, 388 clips, old size was 580GB, new size 412GB and it would take 20+ hours to complete.

After 4 hours the process failed as I ran out of space on my target drive. It had copied almost 1.6TB. It looked like it wanted to copy 5-6TB. So the size of the project 412GB was actually totally off.

Then I was told about ResolveCollect in this thread, a 3rd party app that apparently did a better job at that (allows you to resume and shows a progress of all the files it copies). Not cheap at CAN$100, but I was under a tight deadline so I bought it. I ran that and it copied the project in 4 hours. Total size 1.78TB. I was happy.

I gave the drive to my colourist only to find out none of the multi-cam had been copied! Checking with Nikolai, the ResolveCollect creator, he informed me I needed to flatten all the multi-cam clips. No indication in the manual about that. That was a huge task since 2/3 of the film was shot on multi-cam. So I had to manually flatten all those clips. The problem is you lose all attributes when you flatten a multi-cam clip. Any resizing, rotation, stabilization etc. is lost. So I had to do some, one at a time as I had some clip with specific parameters I could copy attributes, flatten the clip and paste back the attributes but that does not copy the stabilization. I also discovered there is no easy way to see if a clip was stabilized in DR. Unless I'm missing something the button looks the same whether stabilization was applied or not. Some shots I knew didn't have unique attributes I was able to flatten grouped up and paste the attributes. As you imagine,a huge and painstaking task to undertake when pressed for time. 

With the multi-cam flattened, I run ResolveCollect, this time it tells me it needs 5.6TB of space to copy all the data. That seemed about right given the multi-cam clips made up about 2/3 of the show and the first time I ran it it came to 1.76TB. Also that was inline with what Media Management wanted to copy in the first place.

I had a newly formatted 6TB drive attached but the copy button in ResolveCollect was greyed out. I emailed Nikolai right away but as he is based in Sweden I believe (I'm on the West Coast) it was the middle of the night for him and I was not going to hear from him soon. After rebooting and trying to troubleshoot this to no avail, I decided to try running Media Management again as I needed to start this process asap to get the data ready for the colourist the next morning (I had already lost one precious day due to missing multi-cam files).

This time Media Management tells me a much smaller new size 980GB (which was accurate), using the same settings as the first time I tried, it now said a little over 2 hours to do! So with flattened multi-cam clips, the project took 2 hours and was a little under 1TB (with non flattened multi-cam 20 + hours and almost 6TB)

Colourist did not find any issues apart from a green screen shot that did not translate well.

Takeaway / note to self 

Both Media Management and ResolveCollect don't handle multi-cam well at all. Knowing that, I will flatten any multi-cam much earlier in my editing process in the future. 

I did not need ResolveCollect.

When checking if a project will load all the assets, remove all data drives and keep only the target drive. I had done that but multi-cam files were showing as there but the files they refer to were missing. They are nestled in and I should have clicked on them to reveal them and would have seen the red icons. 

PS Nikolai (from ResolveCollect) was pretty swift to get back to me and try to help at first but I never heard back from him as to why the copy button was greyed out despite having sufficient space.

After such a bad experience I know you’ve learned your lesson, so I’m definitely not wanting to rub it in.
But anybody else should take one lesson here: test your workflow in every detail before commiting to a serious job!

Both Media Management and ResolveCollect don't handle multi-cam well at all. Knowing that, I will flatten any multi-cam much earlier in my editing process in the future. 

That's one of those things that's extremely obvious to me, because obviously the colorist only needs the shot used in the final... not all the different angles in the other shots. But I'll add it to my list of how to optimize Media Management.

After such a bad experience I know you’ve learned your lesson, so I’m definitely not wanting to rub it in. But anybody else should take one lesson here: test your workflow in every detail before commiting to a serious job!

100% -- very wise advice. But sometimes, the best lessons you can possibly learn are the painful ones. After that, you can say, "OK, I won't ever do that again." As a pal of mine likes to say:

Good judgement comes from experience.
Experience comes from bad judgment.
Experience is that thing you get after you need it.
And it takes years of experience to get years of experience.

After such a bad experience I know you’ve learned your lesson, so I’m definitely not wanting to rub it in.
But anybody else should take one lesson here: test your workflow in every detail before commiting to a serious job!

For sure but on indie micro budget projects you don’t know who the colorist is going to be and in fact I was originally going to do it.

For sure but on indie micro budget projects you don’t know who the colorist is going to be and in fact I was originally going to do it.

Even if you were going to do the color, there are some fairly rigid rules for post workflow that everybody has to follow. If you don't, you're going to run into some trainwrecks for conform in any color software (Baselight, Lustre, Nucoda, Mistika, Resolve, etc.). There's no way to get around it -- organization, file codecs, plotting out VFX and speed changes, figuring out titles... all this stuff is really crucial. Especially for indies, where if you run into major problems, there's going to be no money left to fix it.

No idea if someone already mentioned that, but yes you can resume after stopping - kinda..

- put all the clips you want to consolidate into one bin, sort them alphabetically by clip name

- do the media management for selected bin!! Resolve will run through them in alphabetical order

- when you stop, just move all the finished clips into a second bin. then start a new consolidate operation for the clips left

[/quote] Even if you were going to do the color, there are some fairly rigid rules for post workflow that everybody has to follow. If you don't, you're going to run into some trainwrecks for conform in any color software (Baselight, Lustre, Nucoda, Mistika, Resolve, etc.). There's no way to get around it -- organization, file codecs, plotting out VFX and speed changes, figuring out titles... all this stuff is really crucial. Especially for indies, where if you run into major problems, there's going to be no money left to fix it.[/quote]

In 30 years working in video games and post audio I'm all about being efficiency and organization. I've put that on my list of R&D as I do not know much about that aspect of video post since until recently I had only done shorts where I do everything myself. I did not think there would be much workflow issues if everything is done in Resolve.