Blackmagic Proxy and Color Management not working

I just tried out BM Proxy and had a test project in 18. I was scrubbing through the project as the proxy generator happened in the background, loved it. Downside is as clips completed, it seemed to break color management and everything looked super flat.

I was using .braw files shot on ursa mini using the BM color managed settings default to SDR and rec709 output. No settings I changed seemed to fix it.

Forgive me for possibly hijacking this thread, but let me say that I will test this also, and report back.

But, er.....where is the proxy generator? I cannot locate the install file/download.

Forgive me for possibly hijacking this thread, but let me say that I will test this also, and report back.

But, er.....where is the proxy generator? I cannot locate the install file/download.

It's installed with the same package that installs davinci resolve studio 18. It's a separate app called Blackmagic Proxy Generator

Yup.

viewtopic.php?f=36&t=158718

This is a problem I reported two versions ago and BMD hasn't prioritized it to fix yet. Proxies of Blackmagic raw files are not handled correctly in Blackmagic Color Managed mode. Other file formats work fine.

Generating proxy files using Resolve 18 public beta will only deliver correct proxies if you generate them with color management off (e.g. DaVinci YRGB in project "color management" settings). The Blackmagic Proxy Generator app has no color settings and will generate proxies that look virtually like the original raw files.

To use the proxies in Resolve 18 public beta, you'll get best (but still not correct) results by

1.) turning color management off in project settings, and
2.) setting the input LUT for all of your Blackmagic raw files to the Blackmagic color space that best matches your camera. You can do this either in project settings or to individual clips in the media pool.

It still isn't anywhere near as good looking as the Blackmagic raw files in color managed mode with no LUT.

What's happening is that the proxies from Blackmagic raw files are shifted, clipped, or compressed when you view the scopes on the color page. I hope Blackmagic will address this soon because it really messes up the proxy workflow if you have footage from Blackmagic and other camera types.

Oh, the irony!

The Proxy Generator is respecting the sidecars, so I'd replace those with something that appeals to you.
But the problem exists for all other codecs.

The Proxy Generator is respecting the sidecars...

But of course, Blackmagic Raw files have no sidecars.

But of course, Blackmagic Raw files have no sidecars.

You can create them manually if necessary, they are just simple json text files.

Or in the RAW tab of DR.

Or in the RAW tab of DR.

Sure, but this is very tedious. Since problem with proxies as I understand is the colorspace issue, it is pretty easy to auto-populate all folders with braw files with sidecar files that force just the specific colorspace and gamma output desired for proxies. Since sidecar overrides are selective, they can contain just the modifications that are needed.

You can make one to your taste (as far as the parameters in the RAW tab go) and copy it to all your source clips.

I've done some extensive testing of proxies for .braw files today and found that there seems to be improvement to proxy CREATION, but Resolve 18 still isn't MANAGING color properly for those proxies.

Proxies created with RCM on or off now look the same as the .braw files. - fixed
When RCM is off or bypassed the waveforms of proxies match those of the .braw files.
When RCM is on proxy waveforms don't match those of the .braw waveforms:

Gen5 Color Science displays a flatter proxy image than .braw,
Gen4 Color Science displays a more saturated proxy image than .braw.
Proxies of non-Blackmagic files (e.g. .MXF) match the originals.

Since the .MXF files are not raw images it may not be a valid comparison. However, it appears that Resolve Color Management is not keeping track of how it processes .braw images when using their proxies.

I was not able to get the sidecar files to work with proxies.

With Public Beta 18.04b there's a new wrinkle in the .braw proxy saga. I've noticed that colors are now better - though still not quite right - in proxies created in Resolve. Whether Resolve Color Management is on or off at the time of Proxy generation, the colors appear slightly oversaturated and the parade scopes show lower dynamic range when played back in Resolve with Color Management on.

But... proxies created in the separate Blackmagic Proxy Generator app exhibit considerably compressed dynamic range (confirmed by the parade scopes), just like ones created in Resolve before the 18.04b.

I've been experimenting with v18 beta 5 today and yeah, proxy files from BRAW in RCM projects are still not working correctly. The closest result I was able to get with proxy files created inside Resolve in an RCM project was only if the Project level raw decode settings for BRAW were set this way:

CleanShot 2022-06-25 at 16.35.57.png (94.23 KiB) Viewed 6008 times

And even then when "prefer proxies" was enabled the image is shifted magenta and slightly more saturated. But the contrast at least matched to when "prefer proxies" was turned off.

To get "optimized media" to look correct requires instead setting the gamma in raw project settings option to linear.

Ideally, these settings should not matter at all. Just as with the BRAW source files where gamma + gamut don’t need to be specifically set and are handled automatically, it should be the same with proxy and optimized media files created in RCM mode. It should just work regardless of the raw setting, camera metadata and sidecar file data, the same as RCM works with the source BRAW file.

The proxy generator app is a whole separate issue since that will create files based on the BRAW file metadata, meaning the metadata in the BRAW file itself or a sidecar taking precedence. Getting it to work would likely mean creating sidecar files to meet the settings that RCM expects (whenever that ever gets fixed....)

You can make one to your taste (as far as the parameters in the RAW tab go) and copy it to all your source clips.

I've posted info about editing sidecar files, including a ZSH script for quickly cloning a sidecar file to a folder full of BRAW clips, here: https://forum.blackmagicdesign.com/viewtopic.php?f=2&t=137011&hilit=+zsh&start=150#p848396

That sounds about right. I saw no difference in my proxies using BMD color space and BMD Film Gamma (as shown) compared to your settings. I don't know the difference between BMD and BMD Wide Gamut Color Space, but your Custom gamut settings are identical to the BMD Film Gamma settings.

Thanks for confirming.

The setting I used is different. Note that use "video black level" is checked. This was necessary to get the contrast to match. And the gamut set to Davinci Wide Gamut doesn't get the saturation and hue perfect, but the result is closer than using the Gen 4 gamut.

In any case, BMD needs to fix it and get this right with their own codec. The project level raw setting options shouldn't affect RCM making proxies. It should simply work correctly by default.

This is still broken in v18 beta 6. No change from v18 beta 5.

This is now fixed in the full release of v18 today! Awesome! Great work BMD!

I'm not seeing any change in the full release of 18. Still same symptoms that you can see on the color scopes: Resolve expands the dynamic range of proxies beyond that of the raw files.

I'm not seeing that issue hereon the full release of Resolve 18 Studio.

What's your source media and what are your project settings?

Changes made to the RAW settings of a clip in a timeline aren't figured into the Proxy creation.

Not loving that, but...there it is.

My source media is a .braw file from Pocket Cinema 6K. Project settings for camera raw are shown above (viewtopic.php?f=21&t=158626#p861202). Color Management is on (DaVinci YRGB Color Managed); no LUTs. No changes have been made to clips in the timeline or anywhere. Everything is straight out of the camera.

Changes made to the RAW settings of a clip in a timeline aren't figured into the Proxy creation. Not loving that, but...there it is.

The data in the proxy files is already YCbCr, it's not "raw" anymore. With the variables involved in when/how the proxy was created (remember they can be rendered in other apps and linked), there isn't any straightforward way to have the proxy file reflect raw tab changes unless the proxy was re-rendered every single time the raw tab attributes were changed.
If making proxies for performance reasons working from raw source files, the render cache is the one to use. The cache for the clip still has to be re-rendered anytime the raw tab values are changed, but at least that happens automatically.

My source media is a .braw file from Pocket Cinema 6K. Project settings for camera raw are shown above (viewtopic.php?f=21&t=158626#p861202). Color Management is on (DaVinci YRGB Color Managed); no LUTs. No changes have been made to clips in the timeline or anywhere. Everything is straight out of the camera.

To get proxies from BRAW source clips that will work as expected in RCM in v18, the Project setting for BRAW needs to be set to "Blackmagic RAW default" before the proxies are generated.

CleanShot 2022-07-22 at 19.25.59.png (45.08 KiB) Viewed 5304 times

Changing to Blackmagic Raw Default seems to have done the trick, although I don't understand why camera metadata doesn't work. I couldn't find anything in a search of the July 2022 manual that explained it.

Can anyone explain how or why Blackmagic Raw Default differs from Camera Metadata if I'm using a Blackmagic camera and a .braw file format?

Thanks!

there isn't any straightforward way to have the proxy file reflect raw tab changes unless the proxy was re-rendered every single time the raw tab attributes were changed.

One way that could work is to be able to render proxies from the timeline instances, so the changes already implemented are incorporated.

Right now, we can only generate proxies from bins, so those changes are (understandably) ignored.

Changing to Blackmagic Raw Default seems to have done the trick, although I don't understand why camera metadata doesn't work. I couldn't find anything in a search of the July 2022 manual that explained it.

Can anyone explain how or why Blackmagic Raw Default differs from Camera Metadata if I'm using a Blackmagic camera and a .braw file format?

Thanks!

Camera metadata can be set to use an external LUT, or be set to BMD Video rather than BMD Film. The Blackmagic Raw default will ensure that no LUT is applied and that the file is decoded to BMD Film which appears to be what v18 is using in RCM when color managing from the proxy file.

there isn't any straightforward way to have the proxy file reflect raw tab changes unless the proxy was re-rendered every single time the raw tab attributes were changed.

One way that could work is to be able to render proxies from the timeline instances, so the changes already implemented are incorporated.

Right now, we can only generate proxies from bins, so those changes are (understandably) ignored.

Respectfully, that simply would not work since the same clip could appear in multiple timelines, and even multiple grade versions, every one of those with potentially different raw tab settings. The only logically feasible method for creating and color managing proxies is from bin source clips (as Resolve currently does), rather than from timeline instances.
If you are using proxy files for reasons of improved performance, then really what you should be using is the cache instead. That will render per timeline, per instance and update when changes are made to the raw tab. Or, since BRAW is so efficient anyway, just drop the decode resolution option in the raw master project setting while working.

The only logically feasible method for creating and color managing proxies is from bin source clips (as Resolve currently does), rather than from timeline instances.

We can create Optimized Media from timeline instances. I haven't tested but I assume there's a separate OM clip for each instance.

You think that couldn't work for Proxies as well?

That asked, your other approaches are valid alternatives.

Optimized media is not created per timeline instance. It still only exists per source frame in the media pool, even when the option to create it is triggered from the timeline contextual menu.
You're right though that it is an option when looking to optimize timeline performance and it does appear to adapt display correctly to some (not all) raw tab decode option changes. But it has to be noted that Optimized Media is very different from a proxy file. It's created per frame, not per whole source clip, it can't be user managed as individual files at the OS file system level (Resolve handles it all internally and stores the frames in with the cache) and it can't be worked from without the source media being online — all in contrast with how proxy files work and what they were designed to do.

Currently in the full release of Resolve 18 I see that Optimized media made from BRAW source clips doesn't display correctly in an RCM project when the BRAW decode is set to "Blackmagic RAW default". It does display correctly when set to "project".

Seems like getting better hardware is the best solution to BRAW playback issues.

Seems like getting better hardware is the best solution to BRAW playback issues.

I hope that is not the impression you got from what I wrote. As I noted earlier in the thread, BRAW already performs so well that optimized/proxy media generally isn't needed for performance reasons. If you are getting dropped frames, simply lowering the decode resolution or turning on timeline proxy mode should do the trick on even an underpowered machine. It's also a good idea to check the drive and connection. Often I see editors having performance issues that have nothing to do with the power of their computer, and are instead the result of a slow drive and/or connection.

Seems like getting better hardware is the best solution to BRAW playback issues.

Often I see editors having performance issues that have nothing to do with the power of their computer, and are instead the result of a slow drive and/or connection.

Yup, I personally would like to get proxies working so I can more easily send media to a remote editor or even myself when I'm not in the office connected to my main server.

Yup, I personally would like to get proxies working so I can more easily send media to a remote editor or even myself when I'm not in the office connected to my main server.

For standard unmanaged projects, proxies have worked with BRAW since the first Resolve release with proxy support (I forget how many versions ago that actually was, somewhere after 16 IIRC). Since the first official (non-beta) release of v18, proxies files made from BRAW now work with the correct transfer function + gamut to display as expected in Resolve Color Management projects as well.