Blackmagic Forum

I've been working on a workflow for removing the FPN from scenes that are not under exposed to begin with. You can view the various other threads on this topic.

I need to hit the sack and will provide more details and examples tomorrow. Bottom line is that I'm seeing improvements to the point where there FPN is essentially invisible for properly exposed scenes and reduced for highly pushed scenes (perhaps more tweaks can provide further improvement here).

In a nutshell I did this:

1) Filmed an out of focus gray card with even 5600k light. I would actually like to redo this with even more even lighting, but my capture is good enough for this test.

2) Converted 100 frames of this footage to 16-bit DPX files using Resolve

3) Loaded 20 files at a time into photoshop layers, converted them to smart objects, and applied 'mean' as the stacking method. I had to break them into 20 file chunks to have enough available RAM.

4) Rasterized these files back down into 16-bit tiff files

5) Took my new files and did steps (3-4) with them again to create a single file that is the average of all sources. Ideally I would do this with even more frames.

The cool thing is that if you stretch the colors in this file, you can really see the FPN pattern of your sensor (will post tomorrow).

6) Now back in resolve I create a layer node with the footage I want to improve and the noise.tiff file. The merging mode is set to subtract.

7) The footage is now too dark, so create a sequential node and apply offset to bring it back up where you want it to be. We are basically subtracting the data from the gray card (which should be very even) plus the fixed pattern from the sensor.

8) Grade as usual

The cool thing is that the pattern seems stable. I'm applying it to footage that I took days ago with nice results.

I also tried this with the lens cap on instead of the gray card without good results. I think that there aren't enough bits down in the deepest blacks to represent what we need.

Ideally this would all be done in camera, but after you have the noise.tiff file it is just a few clicks to get things set. I'm looking forward to seeing if this works in the long run, along with posting samples tomorrow.

Sounds very interesting! I look forward to seeing the samples

Looking into flat field correction, which is essentially what we are doing, there are two parts to be corrected, pixel gain fpn and dark current.
http://en.m.wikipedia.org/wiki/Flat-field_correction

Your method covers the first, though an 18% grey card metered at exposure may not be the best.

The dark current fpn also needs to be accounted for.

Andrew Hunter wrote:Looking into flat field correction, which is essentially what we are doing, there are two parts to be corrected, pixel gain fpn and dark current.
http://en.m.wikipedia.org/wiki/Flat-field_correction

Your method covers the first, though an 18% grey card metered at exposure may not be the best.

The dark current fpn also needs to be accounted for.

Interesting, thank you for the link. I see that it says to do median stacking instead of mean stacking. Perhaps I'll give that a shot and see if the results are any different. Doing some more tests now and the results are very nice.

This doesn't help tremendously with the column bands that show up when pushing shadows that are under exposed. Perhaps that needs the dark current fix as you mentioned. That is trickier since it varies with other variables. Fortunately for me I don't need to shoot in low light often

Black shading is dark current fpn removal in camera, most cameras allow you to update this in the field. The cool thing is DNG allows for the FPN removal frame to be deferred to the post-debayer part of raw processing.

I can see why pixel gain fpn would generally be done in the factory, as the uniformity of the light source is important. Sticking a lens cap on and doing a black shade is much easier in the field. Dark current is also the most sensitive to changes in temperature.

To remove the optics from the equation, you could tape a heavy diffusion over the lens mount and shine a bright light into it, combined with ND filters, to get a better gain source image. I've never done this, so just speculating.

The upside is that this is something BMD could release in a firmware update

Andrew Hunter wrote:Black shading is dark current fpn removal in camera, most cameras allow you to update this in the field. The cool thing is DNG allows for the FPN removal frame to be deferred to the post-debayer part of raw processing.

I can see why pixel gain fpn would generally be done in the factory, as the uniformity of the light source is important. Sticking a lens cap on and doing a black shade is much easier in the field. Dark current is also the most sensitive to changes in temperature.

To remove the optics from the equation, you could tape a heavy diffusion over the lens mount and shine a bright light into it, combined with ND filters, to get a better gain source image. I've never done this, so just speculating.

The upside is that this is something BMD could release in a firmware update

Perhaps the camera hardware doesn't have enough frame buffer or processing power to apply these corrections with. If I can get good enough results in post with a few clicks then I'm happy. It's already not horrible to begin with.

First still sample. These are 100% crops of a test image. Sorry but I cannot post the entire image.

This is the field that I am subtracting with:
https://www.dropbox.com/s/x0bblnry647ro ... educer.tif

Here is the original grade:
https://www.dropbox.com/s/3stwfeugdcjiy ... _grade.png

Extreme sharpening is applied to show the FPN better in a still image:
https://www.dropbox.com/s/tp2jcg4mmas9e ... harpen.png

Here is the fixed image:
https://www.dropbox.com/s/4urnir7499lwkpx/fixed.png

Fixed image with extreme sharpening applied:
https://www.dropbox.com/s/o0buj7uxuzz6t ... harpen.png

Not bad for a first result. I'm sure that making the noise_reducer at different exposure levels or applying a median stack instead of mean stack could improve things also. When I have time

It's easier to see in motion than these stills. I recommend loading two image pairs up in your browser or photoshop and swapping between them.

Looks like it's cleaning up what you guys are seeing, very interesting.

Following along, looking forward to seeing what your lowlight tests bring.

Just thought, it would also be informative to check if one person's grey card works for the next. I'm thinking no but then again... if it does work then you could just build a DRX for resolve and distribute that.

Kholi wrote:Looks like it's cleaning up what you guys are seeing, very interesting.

Following along, looking forward to seeing what your lowlight tests bring.

Yeah I'm really optimistic about this, at least for how I use the camera. Getting some fantastic images out of this baby.

Scott,

I'd also make sure that you do all the math in linear (1.0) gamma so you don't get any gamma induced errors in the subtraction.

I don't have photoshop and gimp is only 8bits. I wonder if it's possible to create a flat field correction in resolve directly?

http://blog.patdavid.net/2013/05/noise- ... ian_6.html

There is a quick way to do this, export stills from resolve and run though imagemagick's convert utility.

Andrew Hunter wrote:Scott,
I'd also make sure that you do all the math in linear (1.0) gamma so you don't get any gamma induced errors in the subtraction.

Yes I was thinking that this would be important to do. However I'm not sure how to convert the input to linear gamma? The 4K image appears non-linear out the gate. I'm sure that a curve could be created but I'm not sure how to go about making this happen and verifying the results. Any ideas?

One way that might work would be to capture light at say 10 different f-stops and then create a curve using these captures as a reference to how linear we are. Then the inverse curve would also need to be made. It would be easier if Blackmagic had curves for us

Andrew Hunter wrote:I don't have photoshop and gimp is only 8bits. I wonder if it's possible to create a flat field correction in resolve directly?

I think you could average a bunch of frames together by putting them on layers and then save the final result. I don't think you could do median however, but I'm not sure how important that is.

Scott Pultz wrote:
Yes I was thinking that this would be important to do. However I'm not sure how to convert the input to linear gamma? The 4K image appears non-linear out the gate. I'm sure that a curve could be created but I'm not sure how to go about making this happen and verifying the results. Any ideas?

BMD supplies a BMD film to rec709 lut and a rec709 to linear invertable lut, so it's a matter of doing the fpn subtraction after the conversion to rec709 but before any further processing.

BMD Film -> rec709 -> linear -> FPN -> invert linear -> any other corrections.

If it was raw, you could go directly to linear to begin with.

Andrew Hunter wrote:

Scott Pultz wrote:
Yes I was thinking that this would be important to do. However I'm not sure how to convert the input to linear gamma? The 4K image appears non-linear out the gate. I'm sure that a curve could be created but I'm not sure how to go about making this happen and verifying the results. Any ideas?

BMD supplies a BMD film to rec709 lut and a rec709 to linear invertable lut, so it's a matter of doing the fpn subtraction after the conversion to rec709 but before any further processing.

BMD Film -> rec709 -> linear -> FPN -> invert linear -> any other corrections.

If it was raw, you could go directly to linear to begin with.

This is close, but then I'm stuck in rec709 at the end rather than where I started. I want to end up in BMD film for other LUTs that I use which depend on this. Anybody know how to generate an inverse LUT?

Another problem is that so far the BMD Film -> rec709 does not work properly with the BMPC as the output color is different than the BMCC. It is less flat with more saturation.

Have they released an updated LUT for the 4K?

Scott Pultz wrote:This is close, but then I'm stuck in rec709 at the end rather than where I started. I want to end up in BMD film for other LUTs that I use which depend on this. Anybody know how to generate an inverse LUT?

Have they released an updated LUT for the 4K?

1D luts are invertable but 3D luts are not. [A-B] is different that [B-A] in matrix math. You can calculate an inverse matrix if you know all the parameters but you can't invert a pre-existing 3d lut.

Kholi and the other beta tests have made mention to it's existence but to my knowledge, it hasn't been publicly distributed yet.

Andrew Hunter wrote:

Scott Pultz wrote:This is close, but then I'm stuck in rec709 at the end rather than where I started. I want to end up in BMD film for other LUTs that I use which depend on this. Anybody know how to generate an inverse LUT?

Have they released an updated LUT for the 4K?

1D luts are invertable but 3D luts are not. [A-B] is different that [B-A] in matrix math. You can calculate an inverse matrix if you know all the parameters but you can't invert a pre-existing 3d lut.

Kholi and the other beta tests have made mention to it's existence but to my knowledge, it hasn't been publicly distributed yet.

We would probably be better off just having a curve to go to and from linear gamma.

Man, I thought I was a tech nerd. You guys are making my head spin!

Is this all more in regards to the BMPC4K? or for the original BMCC as well?

David J. Lee wrote:Man, I thought I was a tech nerd. You guys are making my head spin!

Is this all more in regards to the BMPC4K? or for the original BMCC as well?

We are targeting the 4K. The good news is that if we figure it out, it will be pretty easy to implement

You should get the same result by using the temporal noise reduction. It is basically a median filter over time aka frame stacking.

Robert Niessner wrote:You should get the same result by using the temporal noise reduction.

From what I understand, temporal noise reduction does the opposite, using motion to detect noise. Fixed pattern noise is by definition.... static.... I am open to correction by anyone more knowledgeable.

FPN calibration is a normal part of camera development, as the sensor will not have a completely linear response. It is likely that the samples used to create the calibration of the sensor differ from the production models use and perhaps the cause of the delay with the 4k shipping.

A simple subtraction is also significantly less computationally expensive than temporal noise reduction, making it's improvements available to more people more easily.

Scott Pultz wrote:We would probably be better off just having a curve to go to and from linear gamma.

Agreed, a 1D lut to get from BMD film gamma to linear gamma would be preferable.

Andrew Hunter wrote:

Robert Niessner wrote:You should get the same result by using the temporal noise reduction.

From what I understand, temporal noise reduction does the opposite, using motion to detect noise. Fixed pattern noise is by definition.... static.... I am open to correction by anyone more knowledgeable.

FPN calibration is a normal part of camera development, as the sensor will not have a completely linear response. It is likely that the samples used to create the calibration of the sensor differ from the production models use and perhaps the cause of the delay with the 4k shipping.

A simple subtraction is also significantly less computationally expensive than temporal noise reduction, making it's improvements available to more people more easily.

You misunderstood what I meant to say. Of course temporal noise reduction won't remove the fixed pattern. But you can use it to get your calibration frame by using the frame stacking of temporal noise reduction. It does the same what you did in Photoshop but without the hassle.

You can also try that in VirtualDub by using the temporal smoother set to extremes - there you can easily see what I mean.

Last night I used double diffused light and diffusion over the lens to produce even fields of light. I started at the clipping point and then went down one f-stop at a time until F22 where my lens wouldn't close down anymore.

I brought these into resolve and used the waveform monitor to produce a curve into the linear gamma space and then an inverse curve to get back to where I started. This was tricky and approximate, obviously a 1D LUT would probably be better.

Anyway the subtraction did not work so well when done in the linear gamma space as the shadows became murky. I'm not sure why yet.

source -> subtract noise -> pushed (looks okay)
source -> linear -> original gamma (looks okay)
source -> linear -> subtract noise (linear) -> original gamma -> pushed (looks terrible)

I was thinking that subtracting these two linear gamma images might crush the bits in the lower end which would make the shadows look bad when pulled up? However 32bit floating point shouldn't have an issue I don't think either.

Is the dark frame also in linear gamma when you do the subtraction?

Andrew Hunter wrote:Is the dark frame also in linear gamma when you do the subtraction?

Yes it was. Maybe there was a problem with how I made the curves or perhaps the inverse curve needs to be 100% accurate which was tough for me to do.

I basically made curves so that the swatches showed up like this in the scope:

0 stops down = 1024
1 stop = 512
2 stop = 256
3 stop = 128
etc

I could also try making a version that goes towards linear gamma but not all of the way as the curve gets really really steep towards the bottom. An actual 1D lut would be a lot better here but I don't have one of course.

Honestly doing the subtraction in BMPC gamma was not all that bad in itself. I could also try it in Rec709 since there are 1D luts for that, but I have no desire to film in Rec709.

I improved my results for removing FPN from properly exposed images by:

1) Made a median based FPN map from 130 flat images about two stops down from clipping.
2) Made a second file that is the average color of the entire median map (all the same color)
3) I use two layer nodes. The first subtracts (1)-(2) resulting in just the FPN pattern
4) Then I subtract (3) from my input source

I would love a way to make this node graph simpler or a way to use a powergrade to make it quicker to add to my clip.

I'm not doing this in linear gamma space and the results are still great making the FPN that I was seeing invisible.

I'm not doing anything to improve noise when pushing underexposed images. Might work on that if I start needing it.

So for those trying to remove FPN from properly exposed clips, this works well so far.

Scott Pultz wrote:I improved my results for removing FPN from properly exposed images by:

1) Made a median based FPN map from 130 flat images about two stops down from clipping.
2) Made a second file that is the average color of the entire median map (all the same color)
3) I use two layer nodes. The first subtracts (1)-(2) resulting in just the FPN pattern
4) Then I subtract (3) from my input source

I would love a way to make this node graph simpler or a way to use a powergrade to make it quicker to add to my clip.

I'm not doing this in linear gamma space and the results are still great making the FPN that I was seeing invisible.

I'm not doing anything to improve noise when pushing underexposed images. Might work on that if I start needing it.

So for those trying to remove FPN from properly exposed clips, this works well so far.

I have been able to simplify this by taking step (3), offsetting it so that there are no negative values and saving this as a DPX. Now I can just subtract this DPX file from my source using a layer node. It lowers the gain slightly but that is something that always gets modified down the line anyway.

Peepin' in for some test results!

Here are two shots that had NR applied as I think it is more obvious when the random noise is removed.

With FPN


Fixed FPN


Open into a tab of your browser or photoshop and switch between them. What do you think?

No luck removing column noise of low light images. Either the results are variable or my algorithm is wrong. I'll probably take some sample images days apart and compare them. If they aren't the same then no luck on that one.

Not sure why it says "The attachment fpn_fixed.jpg is no longer available" but you can open the attachment directly.

Results look good to me. Although I'm still not seeing it as an issue on my end, at some point someone's bound to say something about it and if there's a fix then, bring it on.

Thanks for braining on it, Scott.

Kholi wrote:Results look good to me. Although I'm still not seeing it as an issue on my end, at some point someone's bound to say something about it and if there's a fix then, bring it on.

Thanks for braining on it, Scott.

If you actually have something going on and not just staring at a silent image looking for problems, then it's likely to be invisible as is

Seems to really help. I wish this wasn't necessary though

Looks very interesting! Even without removing the noise columns that's a good solid noise reduction.

Apologies for this surely newbie question, but at the end when there's a workflow to follow, would we just do that once for the camera (unique to each persons camera of course) or does FPN change over time? Basically I wonder, how fixed is the fixed noise?

LostBoyNZ wrote:Looks very interesting! Even without removing the noise columns that's a good solid noise reduction.

Apologies for this surely newbie question, but at the end when there's a workflow to follow, would we just do that once for the camera (unique to each persons camera of course) or does FPN change over time? Basically I wonder, how fixed is the fixed noise?

Good question and I unfortunately don't have the answer for this yet. It's been stable for a week but I don't know if it will be stable months from now. I suspect (and HOPE!) that it is stable and that other cameras have a table in them to do this automatically. The BMCC hardware probably doesn't have enough processing power to accomplish this at 4k among all the other things it needs to do.

I'm also testing some workflows that improve things on a wider ranges of images. Right now I'm subtracting the noise from the input, but I think it needs to be scaled to the input first per the wiki page posted above.

Thanks very much Really looking forward to seeing the final result from all this. I think we'll all owe you a few beers!

LostBoyNZ wrote:Thanks very much Really looking forward to seeing the final result from all this. I think we'll all owe you a few beers!

Well I do plan a trip to NZ eventually

It's a beautiful country Home of Peter Jackson and James Cameron, haha. Or photography wise, Trey Ratcliff.

LostBoyNZ wrote:Apologies for this surely newbie question, but at the end when there's a workflow to follow, would we just do that once for the camera (unique to each persons camera of course) or does FPN change over time? Basically I wonder, how fixed is the fixed noise?

Not such a noob question at all.

From my experience with the cameras from RED, which have a very good FPN canceling algorithm, it doesn't change much over time. But it can change a lot with temperature and with exposure time! RED even allows you to save different noise samples for such changes, so you can adapt quickly. The noise sampling process in the camera can take quite some time even on those cameras, which have probably a lot more computing power than BMs. This proves that the approach of making a median over several images is valid, I suppose they do just the same. And, for sure, it has do be done for every camera, not just one production sample!

Since the BMPC doesn't offer very long exposure times, I think it will be related mainly to temperature. You may need different noise samples for very hot or very cold conditions and one might need to let the camera come up to operating temperature before shooting critical imagery.

Plus, a new firmware version might need a new noise sample.

I had to switch out my body with a new unit, missing the helicoils, and the new sensor has a crazy amount of fixed patern noise, where as the previous unit did not.

Pure speculation but I think there is a lot of variability in the sensors coming from the manufacturer and some line up better with the factory calibration than others.

Here is an example of a diy blackshade:

Camera original:
http://i.imgur.com/gjxOyIv.jpg

With dark frame subtracted:
http://i.imgur.com/urc0wYb.jpg

In particular, look at the difference in the FPN in the out of focus areas of the left hand side. There is a shift in the overall exposure that could be attributed to doing the subtraction not in linear gamma.

This also masks out any hot pixels!


You can create the dark frame either in photoshop, as Scott has explained, or via imagemagick, an open source swiss army knife of image processing.

1) Record about 5-10 seconds of footage @ 4k with the body cap on

2) Render the dark clip out of Resolve as 16bit tiff files

3) Run

Code: Select all

convert *.tif -evaluate-sequence median dark-frame-median.tif

from the directory that has the tifs in it.

This will take a long time

4) Import the median file as an external matte in resolve and set up your node graph like this: http://imgur.com/q8JHudY

I'm sure smarter minds than mine can improve the process.

Andrew that does great! I see a different pattern when I do the lens cap on vs having source light. I think that both may need to be subtracted.

Having fun with my rented Otus this weekend so not spending time on this, but will report back with more I'm sure.

is this about the 4k cam or what ?

Alexander Arndt wrote:is this about the 4k cam or what ?

These techniques apply to any camera, the 4k just seems to have the most problematic calibration from the factory.

for the bmcc i use the settings inside davinci.

Alexander Arndt wrote:for the bmcc i use the settings inside davinci.

Hey Alexander,

What are you trying to say?

Any programmer here that would be up for creating a little tool to "drop DNGs here" -> output TIFF for FPN removal?

Christian Schmeer wrote:Any programmer here that would be up for creating a little tool to "drop DNGs here" -> output TIFF for FPN removal?

+1

Christian Schmeer wrote:Any programmer here that would be up for creating a little tool to "drop DNGs here" -> output TIFF for FPN removal?

+1 here too! haha