DolbyVison Metadata import issues with IMFs

When we import a Dolby Vision IMF (made with the Netflix preset) for QC checking, it takes almost an hour to import the Dolby Vision metadata.

Once finished, when we check the tone mapping settings, and find that the "Tone map using..." metadata settings is defaulting to "Custom" as opposed to "Embedded". Why? We want to see the embedded. So we must change the setting.

When making this change to embedded on the first clip, Resolve starts re-importing the Dolby Vision metadata again. This takes the same amount of time as before.

When it is finally imported, we have to switch the "tone map using..." setting to embedded for every clip on the timeline one at a time. There is no way to do this for the entire timeline at once. Why?

So, today, it took me 30 seconds to export only one shot as a supplemental IMF for a QC fix for Netflix. It then took 30 seconds to import the IMF and make a new timeline using the composition playlist.

It then took 1.5 hours in total to import the Dolby Vision metadata and for us to manually change all the clips to embedded, one at a time, so that we could QC check the tone mapping before uploading the fix.

Is there a better workflow that we should use, or is this just the way it is?

Its very very slow at reading the embedded data. we do it too. I figured the better thing to do, use Metafier to extract the XML data, save them per episode/spot/feature, then apply like you normally apply and xml to the clips..
the advantage is, you can do multiple Terminal/Metafier windows at the same time. so if, like us, we get 9-10 episodes from a season, i can extract 3-4 at a time.. and not tie up resolve for doing other work in the meantime..

Extract Dolby metadata from IMF


Extracting a XML metadata file from an MXF:
(watch out for the / in between the destination and filename. no space between them)

./metafier -e (drag destination folder into terminal)/(filename.xml) (drag source mxf into terminal) hit return

This is very helpful, Marc.

Thanks for the tip. I'll try it on the next one.

I think Davinci starts the import process automatically when we load the IMF. So we should just cancel that, and then proceed with the metafier workflow?

What is the Resolve version?

Also interesting to check- what if you right click on imported IMF and then use Extract DOVI metadata? Slow as well?

We just upped to 18, but last time we did DoVi was in 17. I will need to test it I suppose.

What is the Resolve version?

Also interesting to check- what if you right click on imported IMF and then use Extract DOVI metadata? Slow as well?

Just as slow.. painful..

This is very helpful, Marc.

Thanks for the tip. I'll try it on the next one.

I think Davinci starts the import process automatically when we load the IMF. So we should just cancel that, and then proceed with the metafier workflow?

Hmm, i dont see it doin that automatically.. there's got to be hidden prefs somewhere for IMFs.. see my other post..
viewtopic.php?f=21&t=175299

What is the Resolve version?

Also interesting to check- what if you right click on imported IMF and then use Extract DOVI metadata? Slow as well?

Just as slow.. painful..

I just tried short sample in 17.3 and it was fine, but sample is very short, so not really fair test.
Dolby integration is still far from optimal in Resolve.

Maybe Resolve is actually decoding Jpeg2000 in order to import DOVI metadata. It would be crazy approach, but it's possible.
Check if CPU/GPU load is very high during this extract.

On 8 minutes sample it took few seconds to import DOVI metadata (Resolve 17.3.1 Windows).
I see different issue though. "Gamma" for the imported clip is not set automatically to ST2084 which gives incorrect preview (even if clip is tagged with HDR logo to make it more confusing). You need to overwrite manually.

On 8 minutes sample it took few seconds to import DOVI metadata (Resolve 17.3.1 Windows).
I see different issue though. "Gamma" for the imported clip is not set automatically to ST2084 which gives incorrect preview (even if clip is tagged with HDR logo to make it more confusing). You need to overwrite manually.

How many times have you tried using this clip with embedded XML?
DR creates a cache file for these somewhere.. so yes, it will be quick the 2nd and 3rd times you use it..
I though i figure this out too, with Terminal even, and tried to show a co=worker how long it took.. and it was literally like 10 sec for a full 30 min show.. then i tried a new file i hadn't touched that day.. yeah. i didn't wait the 20 minutes to find out.. if you look somewhere in the usr folder.. you'll find the cached file..

It's new clip, so nothing cached (but maybe remembered after export?).
It should be enough to change file name.
There is really no reason for that process to take more than 5min for 1h if it's coded well.
Maybe you are over network and Resolve is scanning MXF headers doing many separate reads- this could be slow.
Try from SSD/NVME.

Our IMFs do end up on a shared NAS for re-import. We export to local SSD RAID then copy it up to the larger NAS.

I will try testing and see if it matter where the file resides.

Thanks.

There is some caching going on so when you open same clip 2nd time it's way faster. I think it's cached in memory though, not on the disk.
I have around 35 seconds (when not cached) from local 2 drives RAID or from very fast, but 10gig attached drive. This is for 8 minutes source.

Did some testing and this is what I am experiencing.

When I import a 90 minute IMF from my local SSD RAID and create a timeline from the composition playlist, with the import Dolby Vision Project Settings checked, it takes about 60 seconds to import the metadata. This was the automatic part I was talking about. If I create the timeline with this unchecked it still imports the Dolby Vision Metadata. There is no way around this that I can figure out.

Then as we all know, the metadata tone map setting defaults to “Custom”. If we change it to “Original” it displays properly and with no delay. If we change it to “Embedded”, it reloads all of the metadata again and takes about 60 seconds. I did reach out to Dolby last fall, when I first posted this issue here, and they confirmed that we should choose “Original” when using this workflow and that it has to be selected on every clip individually. Which is a limitation of the implementation of this feature by Blackmagic. They said to not use embedded, which should speed things along once it is loaded.

I then imported a supplemental IMF of the same timeline with about 34 new shots. Created a timeline the same exact way and it took the same amount of time to load the metadata as before. The clips tone mapping settings defaulted to “Custom” as expected and we changed to “Original” with immediate results. “Embedded” caused another longer reload.

Now in a new project, I repeated the same above process, but this time importing the IMFs from our QNAP over 10G network.

When creating the first timeline as above, the Importing Dolby Vision Metadata took over an hour.

If I cancel this, I can still open the timeline but I can not change the Tone Mapping to “Original” without it continuing to import the metadata again. Taking as long as before.

After this is completed, when I import a supplemental IMF with 34 new clips, when I create the timeline it takes the same amount of time to import the metadata. Over an hour. Even though it already loaded the metadata for the first IMF now that it is a supplemental on another timeline, it has to do it all over again.

I also tried to import the IMF from the SSD, load the metadata from there, and then relink to the IMF on the QNAP. This worked well for the primary. I did not need to reload the metadata. But when I imported the supplemental IMF from the QNAP (34 new clips) it took about 15 minutes. I would guess that it is only loading the metadata for these new clips, but it still takes longer than if it was on the SSD.

So if we make a supplemental IMF to replace one clip (an updated VFX shot in this case) we can not QC check the tone mapping from the IMF without having to first load the entire films metadata which takes over an hour.

It is not practical to keep the IMFs on the local SSD RAID after the initial export. We move them up to the shared storage for later use or to open in another color room. We need the local RAID for render cache or for exporting other projects as well.

It appears that Resolve has issues with the QNAP. We also have render cache issues when we cache to the QNAP or another network storage. I have posted about this in the past, but the only advice I ever get is cache to a fast, locally attached storage device. Which does me no good when I want to move the project to another room. Which I thought was the entire purpose of a shared database. But I digress.

Thanks for any advice on how to speed things up on the QNAP.

Another way to do it is to have xml metadata extracted during creation of the master and then use this instead of embedded (creates some dangerous situation though of misalignment).

Bob Zelin is helping me out with this issue to see if it is Qnap related. He suggested I also measure how long it takes to import the metadata from the same IMF on an external USB drive.

Here is a quick summary of what I tested today.

90 minute - 500GB Netflix IMF source. Import Dolby Vision Metadata times:

SSD RAID: 1 minute 5 seconds
Disk Speed test - 4222 MB/sec write 4350 MB/sec read

Lacie USB-C hardrive: 26 minutes 47 seconds
Disk Speed test - 80 MB/sec write 85 MB/sec read

QNAP: 63 minutes 39 seconds
Disk Speed test - 900 MB/sec write 1072 MB/sec read

There does not seem to be an obvious correlation between disk speed and how long it takes to import the metadata.

Obviously the SSD RAID is super fast, but the Qnap is second fastest but slowest for import.

Any other ideas? Or does this help confirm something already discussed? Thanks.

Well- for me clearly QNAP specific issue.
Simple Windows share over 10gig is not giving such a difference, so there is something with QNAP which causing this. Probably related to fact that reading metadata from MXF generates many requests (metadata is stored per frame not all in single place). Many requests access always is 'slow' with network storage, but something specific to QNAP takes it into extreme in this case.

Reading will be related to disk speed for sure. Maybe not linear, but QNAP has clear issue here.

Probably related to fact that reading metadata from MXF generates many requests (metadata is stored per frame not all in single place). Many requests access always is 'slow' with network storage, but something specific to QNAP takes it into extreme in this case.

This appears to be the general consensus. The question is, does Davinci really need to read the individual frames in order to import the metadata? Not sure how the metadata is embedded in an IMF.

Thanks for all of your assistance.

Dolby metadata is not inside XMLs for sure.
I'm 99% sure you need to scan whole file. DV metadata is not in single place, but aligned to video frames, so stored throughout the whole video mxf file. You need to go to the end to get all of it.
There are different ways of opening file to read though- you can open once and read all or keep opening and closing every call (frame). Something like this- you need a programmer to answer you question. QNAP people should know it better. Seen similar issue with Blu-ray muxing- without subs it was fine with subs (many single files) it was very slow.