DCP Bit depth and formats

Hey,

I have been working for TV (manly commercials) for many years, needless to say that I did not encounter a request for a DCP very oftenly, even when we had an ad screening in a movie theater they would usually ask for a ProRes422HQ having that they dont use the same projector for the ads.

Lately I have been shooting my own films (as a DOP) and happily enough they are getting many theatrical requests.
Having that in my studio I don't have a projector or a DCP server I usually ask a bigger facility to make the DCP. In Israel I can say that its is really the "Wild west" when it comes to formats.

My questions is:
1. Should DCP's be created only from 12bit DPX or is 10bit enough info to go through the XYZ conversion?
2. Would a Prores4444 file be better than a 10bit DPX? considering the film was mastered on a calibrated Rec709 display.

i make a DCDM from my master timeline directly in Resolve, creaitng a jpeg2000 XYZ sequence that is used without further conversions to maek a DCP by copying the files into a DCP folder structure and adding an XML

by doing this i control the color management, and i have elimnated that weak link

I agree that this would be best.
But some companies are insisting on doing the conversion in-house, having that said i'm getting the feeling they are asking me for 10bit DPX because its lighter, and I would like to know what is the standard protocol.
Same with Prores, I usually get asked for it if they have less time or any other problem.

i do not give them a choice, they have to take my DCDM, it saes them time and trouble, takes responcibality away from them for color managent

i can see nothing but a win for the folks doing the packageing, mkaes me wonder just how professional they are...

not excatly confidence building if getting a DCDM delivered to them wraped with a bow on top is too much of a challange for their little brains to handle

There is nothing wrong with supplying DPX 10bit as Rec.709 or already XYZ TIFF etc. You just have to trust company to perform further conversion.
Resolve own Jpeg2000 encoder is rather simple and slow so if your source is long then it will take time.
Ideally you want 12bit or 16bit uncompressed format (DPX, TIFF, EXR). In the same time you can also use ProRes which is some compromise. If you do use ProRes make sure it's 444 option, not HQ etc. If you want to make files smaller than TIFF or DPX then ProResXQ variant is very good option- this will be very small compromise as quality of XQ is very good (quality will be above jpeg2000 encode itself, so this is what you want).
Your source quality also plays a role. If you started with some 8bit (DSLR, prosumer) formats as source then 10bit DPX is good enough. If you have nice Alexa, RED etc. RAW source then you want to preserve as much quality as possible, so you want ProResXQ or 1 of image sequences.

Your source quality also plays a role. If you started with some 8bit (DSLR, prosumer) formats as source then 10bit DPX is good enough.

I'd add that up to HD 1080, 10 bit has ben the standard for quite a while, but once you move up to higher resolutions, then 12 and 16-bit make more sense.

jPo, CSI

Bit depth and resolution are separate here.
People use 10bit as this is (or at least was) enough to preserve all source information. With modern cameras this may not be always the case, so when you trying to make DCP (which is the best delivery option for wide audience) then better to use 16bit TIFF, jut to avoid any possible compromise.

Also, even if your material came in the resolve as 10bit, the simple fact that:
1) you're doing color to it
2) you will transform to XYZ

it will create extra quantization that can be better held giving a final 16 bit (even if the projector in the field currently can only go up to 12 bits anyway.)

think in this way: you have a perfect ramp that goes from 0 to 1023 in exactly 1023 pixels.
you do a correction that bring the top down by 1/1023: if you look at the ramp once you render a 10 bit, you will se a discontinuity somewhere in the middle, if you render it at 16 bit (and look at a proper 12 bit display) the discontinuity will be gone.

Bit depth and resolution are separate here.

They do eventually become entwined, though. 8-bit was perfectly fine for SD, because the resolution wasn't high enough for quantization to become pronounced enough to be intolerable.
With Walter's math exercise, you need more quantization levels to create a smooth grey-scale with larger bit counts. If you want a nice smooth ramp at 3840x2160, you need more levels... it is also true that the farther you try to spread a small quantization difference out over a large number of pixels, the more noticeable that level change will be. Its just what sampled systems exhibit when there is a discrete set of possible values. Then you need to start dithering with whatever tool the application has offered.

jPo, CSI

Yes, but this is very special case
In real footage either it's HD or UHD bitdepth won't make much of a difference. Real footage has always some grain/noise, so any local area (sky etc) won't so quickly start failing because frame is UHD not HD (and 10bit v 12bit). HDR extended "range" is much more important and requires more bitdepth.

Thanks for everyone's replays, very helpful!
Would you then say that a UHD H264 at 80Mbps will produce more banding artifact then a HD H264 35Mbps file? (given that there are problematic ramps in the video)

Thanks for everyone's replays, very helpful!
Would you then say that a UHD H264 at 80Mbps will produce more banding artifact then a HD H264 35Mbps file? (given that there are problematic ramps in the video)

not sure, you have about half the compression bandwidth per square pixels of image, but what also matter is the high frequency of the image: the higher is the high frequency energy of your image, the more difficult is the compression.

I will say yes, UHD will show more banding I that respect.

the other consideration is the display: do you see the image in a similar environment, or the UHD display has double width? do you habe the same viewing distance, do you cover the same field of view...

and so on...

Would you then say that a UHD H264 at 80Mbps will produce more banding artifact then a HD H264 35Mbps file?

Consider some math. UHD is 4x the pixel count of HD1080. Just a crude estimate suggests that matching the 35Mbps rate should really be in the 140 Mbps range for one-to-one correspondence.

Bitrate reduction would likely contribute more to macro-blocking artifacts and aliasing than conventionally-recognized chroma banding. That is more attributable to coarse quantization error -- 8-bit offering a maximum of 256 possible values, 10-bit-> 1024, 12-bit -> 4096, 16-bit-> 65,536... 2 raised to the power of n...

Look deep, very deep, into raw XAVC or even R3D (~jpg2000) sometime. Not very good, whether its even 12-bit or not.

jPo, CSI

That is more attributable to coarse quantization error?

What do you mean by this?

Also when I was talking about banding, yes I was talking about strong banding, that back in the days we would get on a 8bit ramp, after trying to throw Grain and even adding a color wash to the grey BG (all tricks we used to do when the systems were actually only working in 8bit) it is still too dark and too monochrome to hold up im youtube harsh compression.
At the end of the day I was left with saying to the client that they should or switch to another video service (impossible for musicians) or dont shoot a dark grey ramp with lighting changes that go from dark to darkness...I didnt see even one clip that holds that up in youtube.

There are better ways to minimise banding than adding plain noise/grain, but not a single pro software has such a filter. Shame.

There are better ways to minimise banding than adding plain noise/grain, but not a single pro software has such a filter. Shame.

I tried many ways, if you have specific suggestions I would really love to hear them.
I uploaded to youtube:
Any type of codec I could try some at very high bitrate and then some at exactly what youtube recommends, both methods failed.
Also the BG was a type of grey so I add some color tint to it so I would separate the channels and add some values to them.

If you get banding on youtube then this is normal. Their encodes are low bitrate and far from optimised. Not that much what you can do about it. Adding noise in a standard way just causes master to be more difficult to encode. When yo then encode h264 will re-introduce banding as smooth gradients are one of the first thing which get "simplified". You need at least some intelligence in adding this noise.
There are ways of covering existing banding in 8bit source files or making optimised 8bit masters for further transcoding, but this has nothing to do with Resolve.
Look at this example:



http://madshi.net/madVR/debandComparison.png

These are rather bit fiddly processes for someone who is doing project of his life and wants best possible delivery for every platform (DCP, web etc).
I used similar filters to make best 8bit h264 Blu-ray encodes out of 10bit masters.

It would be nice if services like youtube and vimeo had a pro /paid option not to re-encode your carefully tuned work. Spend ages in the past on finetuning /scripting and finding the perfect combination of a bilion of x264 compression /filtering options( you have to do lots of study and testing) to get rid of banding and other nasties, only to have it wiped out by these services upon upload.
Now it seems only usefull for best quality bluray encoding.
Pitty....
So left are only options directy manipulating the image with stuff like adding little random variation/color/colored noise etc to problematic areas.

Youtube is about numbers not quality, like most things these days. Vimeo offers better quality.

That is more attributable to coarse quantization error?
What do I mean by this?

We have a sampled system in play as a practical technical solution for media transmission. "Digital" does not mean "perfect" or "bullet-proof", or even "accurate."

What it is, is more immune to corruption, but once a numeric value has been determined to represent the characteristics of a pixel, it also gets a kind of inertia, especially with respect to its neighbours. Quantizing is a bit of a black art, and there are several strategies for determining a pixel's defining values. If a scene is organically random, what the value of a "next" pixel might be should be considered to be potentially random. Accurate processing times would be prohibitively long to allow for the high-bandwidth display speeds we currently support. So there are chroma-subsampling algorithms, among others, that allow for "best-guess" and error-checking to get a "better" result, faster. Not necessarily "best." Its an engineering assessment.

Consider the rungs on a ladder. Nice, predictable distance between the rungs. Let's say there are 8 steps. Standard riser height is about 11-12 inches. That's an agreed-upon step distance that humans feel comfortable with. That will yield an 8-foot ladder. Might not get you up on the roof, which might be 16 feet, I don't know, maybe you have a deck? In video, this might be the distance from HD to UHD. WE don't just want to stretch the ladder with its existing 8 rungs... the steps would now be a riser height of 2-feet... we obviously need more steps (extension ladders, etc., but this is now a 16-step unit) to achieve a smooth number of level differences.

As far as bit rate goes, Long-GOP efficiency is only as good as the image complexity. The gross effects of "rolling shutter" with its warped objects is a related side-effect of not taking the whole image into account at any instant. None of these strategies can work without dealing with both spatial and temporal factors and their relationship, simultaneously. Getting a good compression is like trying to work a speed/distance correlation with a fixed/given energy resource, where the fuel consumption (which determines range) is dependent on velocity... in that case, the math is not linear; its power-ordered... twice as fast needs 4x as much fuel, but only halves the time... that sort of thing.

jPo, CSI

The responses to your question here are a bit jumbled

My questions is:
1. Should DCP's be created only from 12bit DPX or is 10bit enough info to go through the XYZ conversion?
2. Would a Prores4444 file be better than a 10bit DPX? considering the film was mastered on a calibrated Rec709 display.

#1
People often get hung up on bit depth. the bit depth required depends on your color space and transfer function.
For a rec709 gamma 2.4 master 10 bits is sufficient. The rec 709 color space is not a wide primary color space so each bit will represent a small quantity of color. The gamma 2.4 tone function also works well in 10 bit.

The DCP standard is in XYZ gamma 2.6 space. The XYZ colorspace is extremely wide, so each code value represent much more color. You need 12bit in XYZ to represent colors normally. The 2.6 gamma is not an issue here.

A 10 bit 709 source is fine to be converted into a 12bit XYZ file.

#2
As far as file format goes either should be a fine master. The Prorez will give you some compression advantages over a DPX. For Image quality there will be no real advantage to either one.

#2
As far as file format goes either should be a fine master. The Prorez will give you some compression advantages over a DPX. For Image quality there will be no real advantage to either one.

If you start with RAW assets (not already a ProRes from ALEXA etc.), 1st generation of ProRes (maybe except XQ mode) creates visible (at 1:1 pixel) softness (noise gets softened, which sometimes is not a bad thing), so when blown over projection this will look different than DPX, which should be more "sharp"/refined.
When you take this and encode further to DCP then difference between ProRes source and DPX will be small, but I think it still may be visible.
If you do go for ProRes use XQ mode- this is very good compromise- smaller than DPX, but still very low compression (and 12bit).
Other ProRes disadvantage is fact that data is stored as YUV, so you are getting unnecessary YUV<->RGB conversion twice and this is never lossless (although nothing to worry about).

#2
As far as file format goes either should be a fine master. The Prorez will give you some compression advantages over a DPX. For Image quality there will be no real advantage to either one.

If you start with RAW assets (not already a ProRes from ALEXA etc.), 1st generation of ProRes (maybe except XQ mode) creates visible (at 1:1 pixel) softness (noise gets softened, which sometimes is not a bad thing), so when blown over projection this will look different than DPX, which should be more "sharp"/refined.
When you take this and encode further to DCP then difference between ProRes source and DPX will be small, but I think it still may be visible.
If you do go for ProRes use XQ mode- this is very good compromise- smaller than DPX, but still very low compression (and 12bit).
Other ProRes disadvantage is fact that data is stored as YUV, so you are getting unnecessary YUV<->RGB conversion twice and this is never lossless (although nothing to worry about).

If you start with RAW assets, most likely you will have to scale to final geometry: the scaling algorithm has more to say in apparent sharpness than the RAW itself

Also:

https://ascmag.com/articles/a-clear-look-at-the-issue-of-resolution

If you start with RAW assets, most likely you will have to scale to final geometry: the scaling algorithm has more to say in apparent sharpness than the RAW itself

Yes, but this is irrelevant as it's separate "issue" to export codec. Same affect will be if you use EXR or DPX.
We're talking about look of final master. DPX will look slightly different "sharper", more refined than ProRes, specially if master has bit of noise. You can't avoid it (maybe except XQ mode), but in the same time it can also be a good side effect.

Also:

https://ascmag.com/articles/a-clear-look-at-the-issue-of-resolution

Nice article. It shows that Arri has something unique in their approach to cameras. It also bit misleading as it should also have CGI content generated at different resolutions as cameras are far from being perfect devices so their K calming is way off (as it was proved easily).
I just don't get how I meant to see different resolution when I'm given just HD version to watch. Of course this is basically going to look about the same for all cameras. What we need is 4K file, not HD.

This is still bit irrelevant to final export. Data will be the same before it hits final "codec". DPX (or other uncompressed format) won't touch it from this point where any other lossy codec will (more or less).

Just got Sony a6300 and their HD recording is no near real HD resolution. You need to use UHD recording and downscale to HD to get "real" HD resolution. There is maybe about "real" 2.5-3K in a6300 UHD recording (even if data is read from 6K sensor).
It's probably also true that higher you go with resolution the more difficult is to actually hit desired resolution, so not sure how many current cameras can actually deliver "real" 4K recordings (also proven by this article). This is also easy to prove by generating CGI 4K content which always look way sharper than any real recording (which is prone to so many technology limits and side effects).