DNG HIGHLIGHTS PROBLEM

When I clip highlights, magenta grain appears in the overexposed part of the image.

This is a problem I usually got ONLY with the Pocket camera.

But a few days ago I bought the Cinema Camera and yesterday I had the same issue.
Could be a problem of the new firmware (1.8.2)? Is it possible to restore/reinstall the old firmware?

Yes you can install any version of the firmware. Could it have something to do with the fact that it's possible to clip just one or two of the RGB channels perhaps? Do you have a screenshot to share?

Here you can download the files I posted 8 months ago:
zenwebvideo.it/download/Dynamic.zip

Now there's the same problem with the Cinema Camera.

Are you using "highlight recovery" option in resolve camera raw? Disable this and the pink highlights should go away. I had the same problem with a clipped highlight


Sent from my iPhone using Tapatalk

I know, but this way I can't recover highlights... The point is that with a BMCC I used 8 months ago this didn't happen. I reinstalled v1.5 firmware, but the magenta grain is still there.

When I clip highlights, magenta grain appears in the overexposed part of the image.

BMPC4K 1.9, the same issue.

Does anyone know where to download old firmare versions?

Until they revamped the support pages there was a link leading to older versions. Now there isn't, but the older versions are still there. You just have to "guess" the correct filename...

http://software.blackmagicdesign.com/Ci ... ws_1.7.zip

This link will download version 1.7, but you can just exchange the version number in the filename for 1.5 or similar, the files are still there.

As for the magenta grain: this was first seen on the Pocket camera, which had the "new, improved debayer algorithm" from the beginning. Later a firmware update brought this same debayering code to the bmcc and as far as i know also to resolve, and it seems that since this time both are also suffering from magenta highlights in special cases. Actually there are rumours that the problem is less pronounced (or even resolved) in newer firmware version, but i can't confirm that since i'm carefully avoiding those situations and hadn't had time to do more testing (it's a busy festival summer...).

Thanks Mac,
v1.9 is better than v1.8.2 (the worse for this problem) but the issue is still there.

I downloaded v1.4 (following your link suggestion) and it works. No more magenta grain.
I can't remember which version I tried in March 2013 but I see that 1.4 is 2013 as well.

Thanks again.
Denis

My BMCC showed the magenta clipped highlights even back on 1.5, I don't think it has anything to do with any recent firmware update.

EDIT: I missed the previous post, good to know this goes away with 1.4. Hopefully it is something they can address in future updates.

Good that 1.4 helps. Anyway I dont think people would like to revert to a 1.4 after 1.9.
As for me, I would shurely like to use up to date FW (1.9+) with clear highlights.

Anyway I dont think people would like to revert to a 1.4 after 1.9.

Sadly that's what i have to do again and again for some shots when i need an external monitor, because the bmPcc lost the ability to output 1080i somewhere between 1.5 and 1.6 Most of the time i can do fine with my kinotehnik loupe, but there are some shots that just won't work without an external monitor, and i can't (don't want to) buy another field monitor just for those cases.

It's a pitty that bmd cuts some functionality while improving others.

I made some other tests and the magenta grain came back again even with v1.4. Does anyone know which firmware version was on march 2013? I hope this isn't an hw problem.

The full set of legacy firmware is always available at:

http://update.blackmagicdesign.com/support

Be sure to enable the Include Legacy checkbox when searching for your product to see all of the previous releases.

Thanks David,
the oldest I found is v1.3.1. Just downloaded but can't try it till tomorrow.
I hope this will solve the problem.

I'll let you know.

Hi David,
even v1.3.1 doesn't solve the problem. Still magenta grain on overexposed highlights.
As far as I can see is an hw problem...

Hi David,
even v1.3.1 doesn't solve the problem. Still magenta grain on overexposed highlights.
As far as I can see is an hw problem...

Dear Denis,
It is the problem with maximum values, most probably in the converter - unless it reads the value from the DNG file and the value is too optimistic. If it is coming from DNG, it is a firmware problem.

I haven't seen this problem in over a year on any of my cameras, shooting any mode (Pocket, 2.5K, 4K), on any firmware. But I also no longer debate on what I let clip in camera.

What I mean's that, using zebras at 85, if I can't hold something at 85 then it's gone, and I don't consider anything above that completely recoverable. I don't use highlight recovery in the tab USUALLY, but I just tried it out and when applying my normal working method (85 zebras, waveform, skin in the 30-40 range) I have no issues with roll-off, clipping, magenta highlights, etc.

On the other side, using IR and solid NDs have removed pretty much any issue I've had with skin and color reproduction.

So, I'm going to conclude that you're using 100 percent zebras and just shifting exposure enough until you don't see them anymore. I may be wrong, though!

I Kholi,
you are right. Using zebra, NDs, etc solves the problem.

But unfourtunately sometimes you can't shoot in perfect conditions. Many times I have to shoot outdoor in sunny days with no artificial lights to lighten dark areas. Hence, since the bmcc is a no low light camera, if I don't overexpose highlights I get to much grain (classic low light grain, I mean) when I shift exposure in post.

Anyway, the point is that with the camera I used last year this didn't happen. I'm trying to understand which is the problem with this one.

Thanks for sharing your experience.

I Kholi,
you are right. Using zebra, NDs, etc solves the problem.

But unfourtunately sometimes you can't shoot in perfect conditions. Many times I have to shoot outdoor in sunny days with no artificial lights to lighten dark areas. Hence, since the bmcc is a no low light camera, if I don't overexpose highlights I get to much grain (classic low light grain, I mean) when I shift exposure in post.

Anyway, the point is that with the camera I used last year this didn't happen. I'm trying to understand which is the problem with this one.

Thanks for sharing your experience.

The problem is most probably the calibration. On one of the samples I have the real maximum is 26193, while the tag reads 27580. Thus it also exhibits magenta highlights.

I just reviewed five days' worth of footage shot with my Pocket Cinema Camera with loads of overexposure, as when shooting a trip due west from Florida to California on the I-10, shooting the sun is inevitable. I don't really see any magenta in the blown out highlights.

When the show is done with scoring and final color work, you will be able to find it on YouTube titled "Cross Country at 14,000 MPH."

The shooting was with a Metabones speed booster on a 17-35 Nikkor and a decent variable ND filter at ISO 800, and when darkness allowed, a Kowa 6mm f/1.8 via a Metabones C to M43 mount. The Kowa, a one inch sensor format lens from the machine vision world (where most camera sensors come from) is spectacular. Its shortcoming is that it has a limited f/stop range, stopping at f/11. Because it has a bulging front lens, it will take a unique little rig to get the variable ND on it. All shooting used Film Color Space, which, with the last Final Cut X upgrade, is EXTREMELY simple to process, as the conversion is built in. Film mode allows a LOT of the overexposure to be tamed down during the color correction pass.

The problem is most probably the calibration. On one of the samples I have the real maximum is 26193, while the tag reads 27580. Thus it also exhibits magenta highlights.

Hi Iliah,
can you please explain what do you mean for "calibration, maximum, tag reads"? Is there a procedure I have to do?

The problem is most probably the calibration. On one of the samples I have the real maximum is 26193, while the tag reads 27580. Thus it also exhibits magenta highlights.

Hi Iliah,
can you please explain what do you mean for "calibration, maximum, tag reads"? Is there a procedure I have to do?

Digital cameras have the maximum raw value that can be reached (highlights). Sometimes it is called "saturation", but it is a bit of misleading term, especially if ISO is raised. The DNG standard calls this value "White Level" and explains it this way: "This tag specifies the fully saturated encoding level for the raw sample values. Saturation is caused either by the sensor itself becoming highly non-linear in response, or by the camera's analog to digital converter clipping."
To see the value of the tag for your camera, download and install the free command-line utility called exiftool sno.phy.queensu.ca/~phil/exiftool
- or one of the GUIs that are listed closer to the bottom of the page. With exiftool, the command line is
exiftool -S -H -WhiteLevel your_dng_filename.dng
and the output is something like
0xc61d WhiteLevel: 27580
This means that the white level in the file is supposed to be 27850.
Now to check the actual white level value, download a free trial of RawDigger from rawdigger.com and open the same dng file. In my case it looks like this:

As you can see, highlights are magentish and the lowest maximum value (marked with green arrowhead) is 26193; it happens to be for the second green channel, G2. By the way, you can check the DNG tag value right in RawDigger - press the EXIF button in the upper left part of the RawDigger screen and scroll down a little.

Digital cameras also use white balance, which is 3 or 4 multipliers applied per respective RGB(G) channels. The multiplier for the green channel(s) is usually 1x, or very close to 1x. For the particular shot, we are interested in "As Shot Neutral" tag, which reads R=0.5085 G=1 B=0.8843. To calculate white balance multipliers from it, take the reciprocal values of the "As Shot Neutral" tag, meaning R is about 2x, B is about 1.13x, and G stays at 1x. This means Rmax will be multiplied by 2x and than limited to the tag value, 27850; similarly, Bmax will also be limited to 27850; but G in fact reaches only 26193. So in the highlights we have "minus green" situation, and the common name for "minus green" is purple.

The discrepancy between the tag value and the factual value for the white level is miscalibration, and it happens a lot.

From here, I can see 3 possible ways out. First, to run all the DNG files through a batch procedure to correct the white level value right in the tag. Second, a feature request to the converter developers - allow for user-supplied calibration (for the white level, at the very least). Third, take the camera to the authorized shop for calibration (may need to repeat the calibration from time to time as the electronic components tend to drift).

Thanks Iliah,
THIS is an answer. The only thing a person from Blackmagic support told me is to use the zebra... and when I asked him (before your reply) about calibration and tags, he told me he had no idea.

Anyway. I downloaded RawDigger and after some tests with different images I noticed (using Exif inside RawDiggger) that in all of them the "White Level" is always 60074 and the "As Shot Neutral" tag is always: 0.629 1 0.7904

As regards this image (you can download it here: http://www.zenwebvideo.it/download/magenta_test.zip), Rmax, Gmax, Bmax and G2max are all 62580 which is higher than 60074. So why is there the magentish highlight? In your example the G2 value was the lowest, hence minus green/purple. In my case 62580 is higher/equal for all of them.

Regarding the batch procedure to correct the white level tag, is there a specific software? I went through RawDigger manual but couldn't find anything.

Anyway, the main question is why this problem now (and not with the camera I used in 2013)? Blackmagic support told me there haven't been hardware changes and I'm using v 1.4 of the firmware (also tried 1.3.1. - v1.2 can't be installed on BMCC).

Dear Denis,

Can you please try this command:
exiftool -v3 -IFD0:WhiteLevel=53000 Magenta_test.dng

Great Iliah,
it works. Magenta grain disappeared.

Could you please explain me the command line and why did you choose 5300? I tried the same value with other images and I didn't get the same result (sometimes grain disappears but with a bigger overexposed area).

I also tried to process a group of images (the same Magenta_test.dng copied 3 times) in a directory (in Exif manual I found that FILE can be a directory as well) but I get overexposed files (even if in Exif white level is 5300).

Thanks

Dear Denis,

> explain me the command line

This command line substitutes the value of the white level (the tag we discussed above) with the value of 53 thousand, 53000.

> why did you choose 5300?

Not 5300, but 53000. Please you open the file in RawDigger and switch off the black level subtraction (press the Black Level button on the lower left corner of the main screen and uncheck the Subtract Black). Next, display the histogram (Window - Histogram). You will see something like
Switch on "Linear X-Axis"

The hump in extreme highlights is non-linear / noisy zone (essentially, the hump for the blown-out zone looking like Gaussian distribution instead of the single value indicates it contains just noise due to pixel non-uniformity), and needs to be trimmed off. You can see it starts somewhere around 53000. We can study this hump in more detail, switching off "Auto" under "Linear X_Axis", setting the "Range:" start (left field controls the start, right field controls the end of the range) to 50000, and decreasing the "Bin Size" (set it to 1, and it will be re-calculated to the minimal possible automatically; 8 in my case):

53000 seems to be a reasonable estimation for the beginning of the hump.

> I tried the same value with other images and I didn't get the same result (sometimes grain disappears but with a bigger overexposed area).

The difference between 60074 and 53000 is less than 0.2 EV (calculation is log2(60074/53000) = 0.18 EV) However, the real maximum value depends on the temperature of the sensor and surrounding electronics, and on sensor fatigue. So you may need either to examine the last dng for each shot to establish the maximum for the shot (the last frame usually exhibits the effect in its strongest), or stay on a safe side, which may be 1/3 EV from the 60074, that is 47700.

> I also tried to process a group of images (the same Magenta_test.dng copied 3 times)

I made a dir on my desktop, with the name "1" and put there 5 copies of the file, adding suffices _01, _02, _03, _04, and _05 to the filenames. Next, I applied the following command, telling exiftool to process all dng files ("-ext dng" stands for all files having dng extension) in that directory (the last parameter "/Users/iliah/Desktop/1/" is just the path to the files, without filenames):
exiftool -IFD0:WhiteLevel=53000 -ext dng /Users/iliah/Desktop/1/
It updated all 5 files. To check, I issued the following command:
exiftool -WhiteLevel -ext dng /Users/iliah/Desktop/1/
The result was that all the files received an updated white level:
======== /Users/iliah/Desktop/1/Magenta_test_01.dng
White Level : 53000
======== /Users/iliah/Desktop/1/Magenta_test_02.dng
White Level : 53000
======== /Users/iliah/Desktop/1/Magenta_test_03.dng
White Level : 53000
======== /Users/iliah/Desktop/1/Magenta_test_04.dng
White Level : 53000
======== /Users/iliah/Desktop/1/Magenta_test_05.dng
White Level : 53000
1 directories scanned
5 image files read

Thanks Iliah,
I'm speechless. No words to thank you for your kindness.

Thanks to your explanation now the problem and the solution are clear. I tried with other images using the histogram as reference and it works. Even the multiple file correction.

Thanks again. If there's something I can do to show you my gratitude, please tell me.

Dear Denis,

Glad it worked. A question, however. Earlier you mentioned that "the point is that with the camera I used last year this didn't happen". Do you still have any dng files from that camera? Can you upload a sample?

Sure Iliah,
you can download them here: http://www.zenwebvideo.it/download/OLD.zip

All images are deliberately overexposed.
I hope you can find something interesting.

Thanks again

Dear Denis,

Sure Iliah,
you can download them here: http://www.zenwebvideo.it/download/OLD.zip

All images are deliberately overexposed.
I hope you can find something interesting.

Many thanks. As you can see there is no hump in the highlights, instead the values form a nice 1-level wide peak. Is it the same camera? Serial number tags are different.

Yes, the images where taken with 2 different cameras.
I don't remember which firmware were, but I tried them all on mine starting from v 1.3.1

Why this difference? Firmware is the only "software part" of the camera, isn't it?

Dear Denis,

Yes, the images where taken with 2 different cameras.
I don't remember which firmware were, but I tried them all on mine starting from v 1.3.1

Why this difference? Firmware is the only "software part" of the camera, isn't it?

This is a bit of a sad thing to discuss. Yes, most of the cases the firmware is the only software part of the camera. But firmware relies on the hardware being in-spec. More, firmware either has some modules for calibrating the hardware (out-of-spec hardware can't be calibrated), or it has not. If it has, it may account for temperature drift, aging drift, and other types of drift, or it may not. The constants for compensation may be correct for the particular sensor sample, or they may not. In short, it is a long story - it involves design decisions, quality assurance, and quality control.

As users, we may ask the manufacturers for a tighter design and manufacturing control loops and better handling of the drifts, which comes with a steep price. Or we may ask them to document the user calibration routine, which to me seems like a much better solution.

Anyway, since they said they had no idea how to solve the problem I'll report them our conversation and your technical informations hoping this could be useful for the next firmware update.

Dear Denis,

On a different subject, can you try FastRawViewer? What I'm interested in is - what fps can you reach if the spacebar is held down?

Iliah,
I just installed it but I can't see where fps is. At the bottom the only label dealing with speed is the one that indicates 0.029 to 0.031s while spacebar is held down. Is it this?

Dear Denis,

Iliah,
At the bottom the only label dealing with speed is the one that indicates 0.029 to 0.031s while spacebar is held down. Is it this?

Yes, thank you; it is about 0.03 secs per frame, but a more accurate way to find the speed is to view a sequence of about 1000 frames and a stopwatch to time it.

Here's the result:
1000 frames in 1:24.6

Dear Denis,

> 1000 frames in 1:24.6

Thank you, it is around 12fps than. What videocard is in your system and what disk if I may ask?

Mac Pro (not the new one)
videocard: ATI Radeon HD 5770 1024 MB
disk: Samsung SSD 840EVO 250 GB
CPU: 2,8 GHz Quad-Core Intel Xeon
RAM: 16 GB 1066 MHz DDR3 ECC

Dear Denis,

Mac Pro (not the new one)
videocard: ATI Radeon HD 5770 1024 MB
disk: Samsung SSD 840EVO 250 GB
CPU: 2,8 GHz Quad-Core Intel Xeon
RAM: 16 GB 1066 MHz DDR3 ECC

Right, thank you. Looks to me like plenty of power. Is it your experience too?

You know, there's the Moore's law

Anyway, I've been shooting video for about 10 years and before I worked for a vocational training center where I designed 3D courses with Maya.
Here's my website: http://www.zenwebvideo.it and my Vimeo channel: http://www.vimeo.com/zenwebvideo/videos

What about you? It seems to me you really know the topic.

Hello,

as suggested by Iliah Borg i have tried exiftool procedure setting the white to 25000 after checking the histogram with RawDigger .

It works very well, now i'm able to use the "Highlight recovery" without magenta grid noise !

Furthermore i use the terminal for a batch file correction with the option -r ( pay attention that with -r will scan the folder and subfolders).
To overwrite the original files use the option -overwrite_original ( pay attention that with this option exiftool will not make a backup copy of your files)

Thank you Iliah Borg for the explanations.

A question:
all my CinemaDNG have white set always to 27580, even with different light conditions, why?

Andrea

Hello,

i just wrote an Applescript Applet as exiftool GUI solution to set the WhiteLevel.
Anyone interested to test it?

Andrea

Hello,

i just wrote an Applescript Applet as exiftool GUI solution to set the WhiteLevel.
Anyone interested to test it?

Andrea

Yes, please!

Hello,

i just wrote an Applescript Applet as exiftool GUI solution to set the WhiteLevel.
Anyone interested to test it?

Andrea

Yes, please!

Hello,
this is the link

http://andreamonzini.com/files/WhiteLevelTool.zip

Instructions in the Readme file.

Andrea

all my CinemaDNG have white set always to 27580, even with different light conditions, why?

Andrea

Dear Andrea,

Sorry for the late answer. White level is set close to the linearity limit in highlights, it does not depend on the light but only on the sensor design. Setting the black level is like saying "this is the mark on the bucket, and if the water is on the mark we consider it to be full". However the is sample variation between sensors, so in certain cases even a conservative setting for white level may be too high. The way to avoid the issue is a combination of tighter quality assurance, quality control, individual sensor calibration at the factory, self-calibration in camera. The cost involved is pretty substantial. If the manufacturer understand the issue they may allow the end user to set/adjust the white and black levels.

Is there a exiftool GUI where I cange change the values?
I found two GUIs, but where not alble to acces the white levels.

I'm not really comfortable with command line stuff.

Is there a exiftool GUI where I cange change the values?
I found two GUIs, but where not alble to acces the white levels.

I'm not really comfortable with command line stuff.

+1

Hey guys, After struggling with the command line stuff in windows I finally found a deccent windows batch GUI for ExifTool called ExifMixer. It's pretty straight forward and seems to do the trick.



First you will need to have the Exif tool in your c/windows path or point the program to where it lives on your computer. the path can be set under the options menu.

Just create a new command in the left pane called 'White level set' or something similar and for the value use this -IFD0:WhiteLevel=26500 -v2

You can change that value to suit your shot. 26500 seems to clip the magenta noise for me in most cases.

you will also need to create an Overwrite original file function with a value of -overwrite_original

This will stop it from creating a duplicate set of footage files.

double click on the white level function and the overwrite function you have just created to add them to the middle pane. Make sure you clear any existing sample functions that are already in there first though.

Now if you drop a directory full of dng sequences on the program window it should run through and modify the exif tags. It's reasonably quick. did a dozen clips in the time it took me to go and make a coffee..

you could also set up a few different White level commands of varying values and include them as presets along with the Overwrite original. They are created and saved in the third (far right) pane.

Anyway, Hope this helps.