Null test for video

What's the best way to perform a null test between two video clips in Resolve? Most importantly, what is a good way to distinguish between a complete null and something close enough that you might not notice (like one pixel being slightly different in one frame)?

To get a quick visual check without going into Fusion, you can add two clips on Edit, one above the other, then change the composite mode of one to Difference.

But I'd normally do it in Fusion, Merge node set to Difference.

For a "good enough" check, I just wave my pointer over the image while looking at the viewer's built-in inspector, to see if the values ever change from 0.0. And/or, set the Viewer to Waveform.

For a perfect check without third-party tools: Probe modifier modifying some property, set to Rectangle covering the whole image, Max, check it returns 0.0.

Or a ColorAnalyser node (Reactor) which can do an analysis and display the max, min and average RGBA values in a given frame, or across a range of frames.

Awesome. So I guess that means the difference blend mode can return negative color values? (A null test would have to, right?)

In a null test you're looking for 0.0 - perfect black.

Basically the Difference mode returns the mathematical difference for each of RGBA. A difference of 0.0 is no difference, so if every pixel is 0.0 in all of RGBA, the images null.

To be exact, the Difference blend mode uses this formula:

Code: Select all

abs(image1 - image2)

Ie for each of RGBA, it returns the absolute value from subtracting the pixel in one image from the same pixel in the other. Meaning no, it won't return negative values - abs always returns positive.

Eg:

Code: Select all

1.2 - 2.0 = -0.8
abs(1.2 - 2.0) = 0.8

For info on all the blend modes in Fusion, I highly recommend Simon Ubsdell's video series "The Basics Of Compositing Theory"

Ah, excellent. Functionally equivalent in terms of checking whether or not there is a difference at all.

And yeah, Simon's channel is excellent. I actually recently put those on in the background while I was working, but I should watch them again. Doesn't he basically make all the blend modes from scratch and then build a delta keyer with just channel booleans or something? Pretty sure he's a wizard.

In Resolve you do it by using Subtract or Difference in editing/composite tab and then also Curves slider to boost end result to be more visible. Very quick to setup.
Objective way:

ffmpeg -i source1 -i source2 -filter_complex psnr -f null -

this will measure PSNR value between 2 videos (in other words difference between them).

And I have another way: you can drag along a reference video, then right-click in the Viewer window on the Edit page and choose Difference, and it will show you the difference between the current timeline and the reference. You can also do this on the Color page. We use this a lot for checking conforms, in case there was a positional change we missed. When the pictures null out, we know we're dead-on perfect.

Marc, is there a way with that method to confirm that there aren't slight differences that would be unrealistic to expect yourself to easily notice? That seems to be a particular strength of Tom's Probe method.

You need to use Curves for this and boost what difference is showing you ( it quickly shows ‘black’ so you think there is no difference). Drag top left corner to the right.
If you want objective measure then use PSNR. Anything pass 60dB is very good. Things like 80+ is single pixels difference. Inf is no difference.

That ffmpeg PSNR method is nice, I wasn't aware of that. Definitely a nice way to quickly see an instant pass/fail on a null test.

I did run into the issue of ffmpeg lacking support for some codecs I wanted to test. Eg I was comparing DPX 16bit float vs EXR 16bit float - which nulls in Fusion - but unfortunately ffmpeg doesn't support DPX EDIT: it doesn't support DPX 16bit float, it does support DPX 16bit int.

Here's a screenshot showing how I'd do a null test in Fusion, in a couple of ways. This is comparing TIFF RGB 16bit int with EXR 16bit float:
(click for larger)

- Firstly, I have the viewer in Quad mode, with two waveform (one RGB, one Y) views, and a Histogram. Using these views we can clearly see - without any numerical analysis - that this is not a null.

- Then for numeric confirmation, I also add Jacob Danell's ColorAnalyser plugin (available on Reactor, Studio only), which after analysing the frame shows us that the highest values - the least nully, to use a technical term, are 0.299 in red, 0.000057 in green, and 0.496 in blue. The ColorAnalyser also helpfully puts two points in the viewer, showing us (one of) the position(s) of the min and max values.

For comparison, here's the ffmpeg PSNR analysis of the same two frames (TIFF RGB 16bit vs EXR 16bit half-float):

Code: Select all

[Parsed_psnr_0 @ 0x7ff8dfc04c80] PSNR r:88.646751 g:88.687500 b:88.208286 average:88.508701 min:88.508701 max:88.508701

This second screenshot shows a perfect null in Fusion. Here comparing DPX 16bit float to EXR 16bit float:

(click for larger)

This is confirmed visually by the flat histogram and waveforms, and objectively by the ColorAnalyser showing Max values of 0.0.

Since when ffmpeg doesn't support DPX?
You mean float DPX which is something rare?

If you comparing different formats (specially float vs. non-float) you can get some difference as it can be down to conversions rounding. You would have to be extra careful with this.
Best is to compare the same formats then you cross any conversion issues.

ffmpeg PSNR filter can work at best in 16bit, so no float support. If you comparing anything float then it's not 100% valid result as you getting conversion to 16bit internally (you can see it if you add -v debug).
If you compare eg. 16bit DPX and 16bit TIFF then you will get inf result.

Yes you're right, it's just 16bit float DPX it doesn't support. 16bit int does work, and comparing EXR 16bit float to DPX 16bit int gives a similar result as comparing TIFF 16bit:

Code: Select all

[Parsed_psnr_0 @ 0x7fe1774077c0] PSNR r:88.646751 g:88.687500 b:88.208286 average:88.508701 min:88.508701 max:88.508701

As you say, these are likely rounding errors, and judging by your earlier remarks, a value of 88 is <1 pixel different anyway.

The ffmpeg docs say that both inputs need to be in the same pixel format, so maybe it's impossible to show a perfect null between codecs where different pixel formats are involved.

Problem is in the lack of float support for PSNR filter. Up to 16bit int everything should work fine.
Best to keep same pixel formats, but it can work with different as well.
If you take simple frame, eg. color bars and export as 8 and 10bit then this most likely will still show inf.
If you take real footage frame from 10bit source then it won't show inf for 8 vs 10bit uncompressed output.
If you convert that frame into 8bit and then do export from this to 8 and 10bit then it should show inf.
You get here into some "dangerous" zone of things happening internally which you are not aware. Same applies to Resolve or Fusion (eg. there may be some internal dithering etc. during specific conversion).

If you compare eg. 16bit DPX and 16bit TIFF then you will get inf result.

Problem is in the lack of float support for PSNR filter. Up to 16bit int everything should work fine.

Yeah, true:

Code: Select all

tomj@Eddie /Volumes/4TB/TestRenders/PSNR $ ffmpeg -hide_banner -i DPX/TestDPX16int00000000.dpx -i TIFF/TestTiff16int00000000.tif -filter_complex psnr -f null -
[dpx_pipe @ 0x7fac7b00a200] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, dpx_pipe, from 'DPX/TestDPX16int00000000.dpx':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: dpx, rgb48be(gbr/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 25 tbn, 30 tbc
[tiff_pipe @ 0x7fac8b011800] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #1, tiff_pipe, from 'TIFF/TestTiff16int00000000.tif':
  Duration: N/A, bitrate: N/A
  Stream #1:0: Video: tiff, rgb48le, 1920x1080, 25 fps, 25 tbr, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 (dpx) -> psnr:main
  Stream #1:0 (tiff) -> psnr:reference
  psnr -> Stream #0:0 (wrapped_avframe)
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: wrapped_avframe, gbrp16le(pc, gbr/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
    Metadata:
      encoder         : Lavc58.134.100 wrapped_avframe
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=1.41x
video:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[Parsed_psnr_0 @ 0x7fac8a525a00] PSNR r:inf g:inf b:inf average:inf min:inf max:inf

Code: Select all

[Parsed_psnr_0 @ 0x7fac8a525a00] PSNR r:inf g:inf b:inf average:inf min:inf max:inf

In older Resolve conversion between uncompressed 4:2:2 10bit (v210) was a perfect example. In theory this should always be inf, but in Resolve 1st conversion itself was already down to eg. 70dB due to chroma sampling algorithm used by Resolve (v210 was going to 32bit float and back to v210). After 10 conversion both uncompressed videos were so different to naked eye, even if we kept exporting uncompressed! It has been changed now, so pure export (without any processing invaded) will keep always inf.

You should never trust any software and verify core things.
Same applies to eg. BM video preview. There is still a dangerous setting which is color matrix for 422 SDI preview. For SD output you can manually set it to Rec.601 or otherwise your SD monitoring may be using Rec.709 which is simply wrong (or maybe it's now set by auto to Rec.601 for SD- never checked). I bet you only 1% Resolve users is aware of this (probably as many uses SD where color accuracy is important).
I'm actually not sure if Rec.2020 is properly used by default for UHD preview, but I "assume" it's. In any case you can set it by yourself in project settings/video monitoring. There are bugs like this and people are affected for years as they never bother to validate it assuming in software like Resolve everything is perfect. It never is.