Blackmagic Forum

The source will be small-guage film scanned 'flat' 4K on a Lasergraphic ScanStation. Output will be 16-bit-per-component DPX image sequence.

The film was shot between 1940 and 1964 on Kodachrome (and a little B/W) by amateurs who didn't always get the exposure right. There is some historically significant 'b-roll' footage I want to be able to market.

The DPX file(s) will likely present bandwidth issues, so I anticipate transcoding to some intermediate format or other. But I note that most of the professional intermediate formats are limited to 12-bit – which, I concede, will likely be adequate for grading most of the shots.

What are your workflow recommendations?

What intermediate format do you recommend, and why?

Maybe this suits:

CineForm Introduction

GoPro CineForm® is a 12-bit, full-frame wavelet compression video codec. It is designed for speed and quality, at the expense of a very high compression size. Image compression is a balance of size, speed and quality, and you can only choose two. CineForm was the first of its type to focus on speed, while supporting higher bit depths for image quality. More recent examples would be Avid DNxHD® and Apple ProRes®, although both divide the image into blocks using DCT. The full frame wavelet as a subject quality advantage over DCTs, so you can compression more without classic ringing or block artifact issues.

Pixel formats supported:

8/10/16-bit YUV 4:2:2 compressed as 10-bit, progressive or interlace
8/10/16-bit RGB 4:4:4 compressed at 12-bit progressive
8/16-bit RGBA 4:4:4:4 compressed at 12-bit progressive
12/16-bit CFA Bayer RAW, log encoded and compressed at 12-bit progressive
Dual channel stereoscopic/3D in any of the above.

Here is some more info on different intermediate codecs
https://blog.frame.io/2017/02/13/compare-50-intermediate-codecs/

If you want to stay in DPX, you can ask for DPX 12 bits or DPX 10 bits: if it is scanned log, there is virtually no difference between 10 bit and 16 given that the material will have lots of grain.

I second 10-bit DPX if you have the space/bandwidth.

10-bit should be fine but only if it is in log format.
Linear requires a higher bit-depth (> 12 bit)

You could also use 16-bit half-float exr-s with compression. If your cpu isn't a weakling you simply trade io throughput with decoder processing. Zip compression is a safe choice but you could also do a few tests and find a suitable level of DWA compression, which produces way smaller files. It isn't mathematically lossless but neither is Cineform/Prores/DNx.

When encoding to exr keep in mind that since it is based on float data type, input data must be linearized to make best use of available code values.

Hendrik Proosa wrote:When encoding to exr keep in mind that since it is based on float data type, input data must be linearized to make best use of available code values.

Linearized data does not make the best use of available code values.
Log does a far better job.

However, for float, the distribution of code values does not make much of a difference.

Cary Knoop wrote:

Hendrik Proosa wrote:When encoding to exr keep in mind that since it is based on float data type, input data must be linearized to make best use of available code values.

Linearized data does not make the best use of available code values.
Log does a far better job.

However, for float, the distribution of code values does not make much of a difference.

Log encoding makes better job for integer storage, not floats. For floats storing log is not beneficial in any way and can be detrimental. If you want I can describe this detrimental effect in detail. Half float storage has limited absolute precision depending on specific value ranges (as opposed to integer storage which has constant absolute precision over whole value range), so compressing lots of data in nonlinear manner can exceed this threshold. How much it affects the image in practice is questionable, but as I said, it has definitely no benefit (unless one considers nonstandard encodings and possible confusion a benefit).

Hendrik Proosa wrote:

Cary Knoop wrote:

Hendrik Proosa wrote:When encoding to exr keep in mind that since it is based on float data type, input data must be linearized to make best use of available code values.

Linearized data does not make the best use of available code values.
Log does a far better job.

However, for float, the distribution of code values does not make much of a difference.

For floats storing log is not beneficial in any way and can be detrimental. If you want I can describe this detrimental effect in detail.

Sure, please go ahead and describe in detail the detrimental effect of using log in floats.

Hendrik, I don't think that the loss of precision in a exl log vs a dpx log does matter if the code value is smaller than both the signal to noise ratio or (in this case) the horrendous grain you will get from the film itself.

In a chain you're bound (as precision) to the bigger tolerance. In this case is not the codec...

Walter, I agree, that it is up to user to make informed decisions. This is why I also suggest lossy compression if applicable.

Cary Knoop wrote:Sure, please go ahead and describe in detail the detrimental effect of using log in floats.

Lets see scene-linear value range of 0.18 to 1.0, medium gray to maximum diffuse reflection. This scene-linear value range 0.18-1.0 is encoded to Alexa LogC (lets use this as an example) normalized value range 0.39101-0.57063
In half-float storage 0.39101 is encoded in bits as sign 0, exponent 01101, mantissa 1001000001
In half-float storage 0.57063 is encoded in bits as sign 0, exponent 01110, mantissa 0010010000

As we can see, exponent changes by one step here. Mantissa for 0.39101 expresses 10bit value 577 (from range 0-1023 that 10 bits can encode) and mantissa for 0.57063 expresses value 144. Their difference is 592 code values (inclusive of 577 and 144, 577...1023 + 0-144) which is a bit more than 9 bits worth.

In float storage, this scene-linear value range 0.18-1.0 when encoded to LogC produces value range 0.39101-0.57063 which in 12 bit integer storage is value range 1601-2336, which is 736 code values. So when storing 12 bit log data as 16 bit half float, we lose 144 code values, they are fused/rounded to adjascent values. Essentially it turns original 12 bit data to data with about ~11 bit precision, and precision issues are worsened the higher on value range we get, because absolute precision is halved for every exponent change (doubled range is covered by same 1024 discrete code values). Fortunately log encoded values don't cross the 1.0 value threshold in floats, so maximum loss is contained to 0.5-1.0 range (but is unevenly distributed inside the range).

But... lets see for example value range 0.8-1.0, a subsection of 0.18-1.0. This scene-linear value range is encoded to Alexa LogC value range 0.54693-0.57063
In half-float storage 0.54693 is encoded in bits as sign 0, exponent 01110, mantissa 0001100000
In half-float storage 0.57063 is encoded in bits as sign 0, exponent 01110, mantissa 0010010000

Going through same methodology (0001100000 == 96, 0010010000 == 144) we see that this range is encoded using 49 code values. From previous 592 code values we get for storing linear value range 0.18-1.0, 49 are used for 0.8-1.0 range. So value range 0.18-0.8 has about 543 code values to use when we stuff log data to half float, when in 12bit precision integer storage and log encoding this range is 1601-2239, 639 code values. So we are losing about a bits worth of data here. But what about the 0.8-1.0 range itself? In integer storage this is 2239...2336 = 98 code values (inclusive). Compared to what is left in float storage, we lose half of them, about exactly one bits worth. Float data absolute precision follows specific sawtooth pattern and this jumpyness can cause banding and all kinds of funky effects for log data.

Whether it is relevant to user is up to user, but this is the possible detrimental effect.

EDIT: had to revise the piece due to stupid error and thus threw out the more sensational wording

I had to revise my text because there was an obvious error, which ofcourse wasn't obvious when I write it first time, classic! Anyway now it should all be correct and if not more, gives an overview of precision loss mechanics when storing log data directly in floats, without linearizing it first. To pull it all together, going from 12 or 16 bit log data to half-floats will lose precision, especially when source is 16 bit. It might sound like stuffing 16 bits into 16 bits results in 16 bits but in reality it is around 11 bits equvalent worth of precision left. How relevant it is... depends on what the input data is. Stuffing 10bit log directly to half floats doesn't lose anything but gains are a bit questionable, compression options of exr can be one direct benefit that dpx does not have.

Cineform at best quality setting made directly from DPX will present no real difference and let you work way easier (it will be about 3-4x smaller). It's 12bit LOG encoded.
I have hard doubt that 12bit LOG will loose anything meaningful compared to 16bit DPX.
Of course another beauty is that you can work with Cineform even on laptop, just by switching resolution to 1/2 or 1/4. CPU needs reduce proportionally, so even 4K files work on laptop with no problem.

Andrew Kolakowski wrote:Cineform at best quality setting made directly from DPX will present no real difference and let you work way easier (it will be about 3-4x smaller). It's 12bit LOG encoded.
I have hard doubt that 12bit LOG will loose anything meaningful compared to 16bit DPX.

16bit linear to 12bit log doesn't lose anything meaningful, and 16bit log to 12bit log doesn't probably lose anything very useful either. Stuffing 12bit log to 16bit half float without linearizing loses one bit of precision on average, so whether it makes a difference is something to decide depending on what the source and whole process is.

After asking the question, after having the reel scanned on a Lasergraphics ScanStation, after loading the resulting 1.6TeraBytes of files onto a fast, Thunderbolt3 SSD...

...I find my question about how to narrow the bandwidth was moot.

Equipment:
iMac Pro 10-core, 64GB RAM, 64X GPU
OWC Envoy Pro EX 2TB Thunderbolt-3 SSD

There's enough bandwidth to handle 24FPS.

I can do everything including [Deliver] without pre-rendering, though I admit I haven't tried throwing a bunch of filters onto clips.

But a couple or three color nodes and Translation stabilizer (to remove the worst of the handheld shake) hasn't caused any slowdowns in playback.

The information in the replies is great, and if I didn't have the SSD, I'd need it.

Thank you all who replied.