Lossless CinemaDNG RAW And Compression

The place for questions about shooting with Blackmagic Cameras.
  • Author
  • Message
Offline

Johnny Harris

  • Posts: 74
  • Joined: Wed Jan 14, 2015 4:29 am

Lossless CinemaDNG RAW And Compression

PostMon May 04, 2015 4:29 pm

Sorry if I'm a bit thick but all the info I could find on this issue is not very clear at all.

How exactly is the compression of Raw working taking into account the following (from the BM Ursa specs):


Storage Rates
4608 x 2592
CinemaDNG RAW 3:1 - 180 MB/s
CinemaDNG RAW 4:1 - 135 MB/s

Recording Formats
Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1


I understand that Raw compression can be akin to zipping a file, so it is lossless. But I don't understand the above even taking that into account:

1) The recording formats cites "Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1" - Does that mean that it's Raw uncompressed, and then the compressed versions 3:1 and 4:1?
OR does that mean that it's Raw in either 3:1 and 4:1.

Not trying to be picky, but grammatically that statement with the comma as in "Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1" denotes 3 separate things.

If it is so, then what's the storage rates for uncompressed raw? You can clearly see the rates for 3:1 and 4:1, but no uncompressed.

2) If the compression is lossless, then why have a 3:1 AND a 4:1 ratio? Wouldn't it make sense to keep it at the 4:1 to get better data rates. I mean lossless is lossless... why work with 180 mb/s when you can work with 130 mb/s and get same result.

It's probably something I'm missing but if anyone here can explain this it would be much appreciated.

Thank you! ;)
Offline

Jules Bushell

  • Posts: 1026
  • Joined: Wed Sep 05, 2012 3:06 am
  • Location: London, England

Re: Lossless CinemaDNG RAW And Compression

PostMon May 04, 2015 5:52 pm

Johnny Harris wrote:Sorry if I'm a bit thick but all the info I could find on this issue is not very clear at all.

How exactly is the compression of Raw working taking into account the following (from the BM Ursa specs):


Storage Rates
4608 x 2592
CinemaDNG RAW 3:1 - 180 MB/s
CinemaDNG RAW 4:1 - 135 MB/s

Recording Formats
Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1


I understand that Raw compression can be akin to zipping a file, so it is lossless. But I don't understand the above even taking that into account:

1) The recording formats cites "Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1" - Does that mean that it's Raw uncompressed, and then the compressed versions 3:1 and 4:1?
OR does that mean that it's Raw in either 3:1 and 4:1.

Not trying to be picky, but grammatically that statement with the comma as in "Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1" denotes 3 separate things.

If it is so, then what's the storage rates for uncompressed raw? You can clearly see the rates for 3:1 and 4:1, but no uncompressed.

2) If the compression is lossless, then why have a 3:1 AND a 4:1 ratio? Wouldn't it make sense to keep it at the 4:1 to get better data rates. I mean lossless is lossless... why work with 180 mb/s when you can work with 130 mb/s and get same result.

It's probably something I'm missing but if anyone here can explain this it would be much appreciated.

Thank you! ;)

This would be visually lossless. So if you could decompress back, it wouldn't be the original uncompressed DNGs. That's why the compression ratios can be high and also different. 4:1 I assume would lose more of the original information than 3:1 but still be visually lossless to the eye when comparing it the original.

For mathematically lossless, meaning you get back the exact original DNGs so that nothing is lost, you'd need compression ratios of the order of 1.5:1, hence lower than visually lossless compression rations.

Jules
Jules Bushell
url: www.nonmultiplexcinema.com
url: www.filmmeansbusiness.com
url: www.blurtheline.co.uk
Offline

Johnny Harris

  • Posts: 74
  • Joined: Wed Jan 14, 2015 4:29 am

Re: Lossless CinemaDNG RAW And Compression

PostMon May 04, 2015 8:35 pm

Jules Bushell wrote:
For mathematically lossless, meaning you get back the exact original DNGs so that nothing is lost, you'd need compression ratios of the order of 1.5:1, hence lower than visually lossless compression rations.

Jules


So to reiterate what you're saying: There are 3 compression modes:
1) Uncompressed
2) Visually lossless 3:1
3) Visually lossless 4:1

This is what is meant by:
Recording Formats
Lossless CinemaDNG RAW, RAW 3:1 and RAW 4:1


Makes sense but then what are the storage rates for (math) lossless?

They're not given for Ursa like they're given for let's say the 4k Production. In the 4k they say that each frame is about 8mb for uncompressed. In the Ursa specs there's no mention of uncompressed and its data rates. All we get is:
Storage Rates
4608 x 2592
CinemaDNG RAW 3:1 - 180 MB/s
CinemaDNG RAW 4:1 - 135 MB/s


In another thread (http://forum.blackmagicdesign.com/viewtopic.php?f=2&t=33575#p199125) which I found a BM staffer says "The Pocket, BM4K, and URSA all use lossless compressed raw at roughly 1.5:1" - but rates for this 1.5:1 are not mentioned in the Ursa specs.

Johnny
Offline

David Hessel

  • Posts: 274
  • Joined: Mon Mar 10, 2014 7:53 pm

Re: Lossless CinemaDNG RAW And Compression

PostMon May 04, 2015 10:36 pm

You are right that they are not given but for the BMCC it is roughly 30% smaller file size for lossless compression so I suspect the same for the URSA. So about 380 MB/s.
David Hessel
Offline

Stephen Ecob

Blackmagic Design

  • Posts: 7
  • Joined: Wed Jan 28, 2015 1:46 am

Re: Lossless CinemaDNG RAW And Compression

PostTue May 05, 2015 12:17 am

Across our camera range we currently support a variety of RAW formats:

* Uncompressed RAW

* Losslessly compressed RAW. This has zero loss of detail but only mild compression - around 35% file size reduction for very blurry / underexposed / overexposed images ranging to around 25% file size reduction for well focused, well exposed, detailed images.

* 3:1 compressed lossy RAW. Some image detail is lost, but not much. The compression is rate controlled to 3:1, so recording capacities of media are very predictable.

* 4:1 compressed lossy RAW. Same as 3:1 but with more compression. The quality is still good, but not as good as 3:1
Offline

CaptainHook

Blackmagic Design

  • Posts: 1440
  • Joined: Wed Aug 22, 2012 4:50 am
  • Location: Melbourne, Australia

Re: Lossless CinemaDNG RAW And Compression

PostTue May 05, 2015 1:45 am

After firmware 2.1, the Cinema Camera was changed from uncompressed RAW, to mathematically lossless compressed RAW.

So now NONE of our cameras with current firmware use uncompressed RAW.

The 3 modes currently offered in URSA are:

• Lossless Compressed RAW
• Visually Lossless 3:1 RAW (lossy)
• Visually Lossless 4:1 RAW (lossy)

You're looking at around 310MB/s for Lossless RAW 4608x2592 @24fps.
http://www.captainhook.co.nz/blackmagic-cinema-camera-lut/

**Any post by me prior to Aug 2014 was before i started working for Blackmagic**
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostTue May 05, 2015 4:10 am

At these data rates for full sensor raw at 24 or 25 fps, I believe you should be able to record safely writing to one Lexar 256GB CFast 2 card, rated up to 450 MB/s. You should even be able to write at 30 fps to a single card theoretically, but with a data rate of about 400 MB/s in practice, you may not want to risk it. No problem recording on dual cards. At higher frame rates, dual recording with visually lossless compression 3:1 or 4:1 is necessary.


Rick Lang
Sent using Tapatalk HD
Rick Lang
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostTue May 05, 2015 4:18 am

Stephen Ecob wrote:... Losslessly compressed RAW. This has zero loss of detail but only mild compression...
around 25% file size reduction for well focused, well exposed, detailed images...


Thanks for this guidance. That compression is less than what I would have thought so good to err on the side of caution when estimating data rates and storage capacities.

Rick Lang
Sent using Tapatalk HD
Rick Lang
Offline

Johnny Harris

  • Posts: 74
  • Joined: Wed Jan 14, 2015 4:29 am

Re: Lossless CinemaDNG RAW And Compression

PostWed May 06, 2015 10:01 pm

rick.lang wrote:At these data rates for full sensor raw at 24 or 25 fps, I believe you should be able to record safely writing to one Lexar 256GB CFast 2 card, rated up to 450 MB/s. You should even be able to write at 30 fps to a single card theoretically, but with a data rate of about 400 MB/s in practice, you may not want to risk it. No problem recording on dual cards. At higher frame rates, dual recording with visually lossless compression 3:1 or 4:1 is necessary.


Rick Lang
Sent using Tapatalk HD


So if I were to use 30fps with 2 cFast cards, this would work out because it would split the data rate between the cards - in much the same way that a RAID 0 writes data across multiple drives?

Data retrieval would have to be done directly from the camera I suppose since an external cFast card reader would only read 1 card.
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostWed May 06, 2015 11:18 pm

Dual recording is better than RAID 0 in the sense that if a card should ever fail, you could attempt to recover the footage in post from one good card; with RAID 0, if you lose one drive, you have lost all your data. Grant Petty I believe pointed out this benefit in a NAB 2015 video. And the audio file is recorded completely on each card so no worries about sound if you lost a card.

You can still use a program like Media Express to pull the folders and files, one card at a time, into your computer for permanent storage. After you do that data capture, then everything looks the same in Resolve or NLE as if you had recorded everything on one card.

Rick Lang
Sent using Tapatalk HD
Rick Lang
Offline

Stephen Ecob

Blackmagic Design

  • Posts: 7
  • Joined: Wed Jan 28, 2015 1:46 am

Re: Lossless CinemaDNG RAW And Compression

PostWed May 06, 2015 11:37 pm

Yes, those are good points Rick.

Johnny, to answer your question:

> Data retrieval would have to be done directly from the camera I suppose since an external cFast card > reader would only read 1 card.

CinemaDNG recordings create a separate file for each frame. For dual card recording even numbered frames go to one card and odd numbered frames go to the other card. If you want to use a single external cFast card reader you can, you read in the even numbered frames from one card and then swap cards and read in the odd numbered frames from the other card. If you lose one card then you can still recover a 15fps recording from the remaining card.
Offline

Scott Stacy

  • Posts: 926
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: Lossless CinemaDNG RAW And Compression

PostThu May 07, 2015 12:26 am

rick.lang wrote:At higher frame rates, dual recording with visually lossless compression 3:1 or 4:1 is necessary.


Hey Rick and Captain Hook ... Am I correct in deducing that dual cards are need for 3:1 raw at 60fps on the mini? Will the current Lexar Cfast cards gobble up data at that frame rate? And, how much recording time does that permit with two 256 cards?

On a data management note, my limited understanding is that with two cards, "data" is being recorded alternating between both cards. How is that data reintegrated? And, regarding on set DIT management, can you use Shotput? Would you just dump one card and then the second into one folder and upon importing into Resolve for dailies would the DNGs from the two cards reintegrate? Maybe I am making this too difficult. Just wondering how I am going to conduct media management when my mini arrives. Thanks.
Scott Stacy, C.S.I.
scott@hppictures.com
Windows 10 version 1803 Build 11/12/18
i7 6950x
2-Titan X
Decklink Pro 4K
Areca 1883 48TB RAID6
Offline

CaptainHook

Blackmagic Design

  • Posts: 1440
  • Joined: Wed Aug 22, 2012 4:50 am
  • Location: Melbourne, Australia

Re: Lossless CinemaDNG RAW And Compression

PostThu May 07, 2015 12:54 am

Scott Stacy wrote:Am I correct in deducing that dual cards are need for 3:1 raw at 60fps on the mini?

It will depend on your card speed, and resolution. 60fps RAW 4:1 @4.6K should be fine to a single card, which is what i shot on the big URSA 4.6K for the demo footage on our site.

Scott Stacy wrote:How is that data reintegrated?

Each alternating frame is recorded to each card and numbered accordingly. Odd numbered frames on one card, even on the other. Because they're sequentially numbered, you can copy both cards into the same folder on your computer and it will be as if they were recorded to a single card from then on, especially as far as software like Resolve is concerned. The audio file is recorded to both cards, for redundancy.

A "trick" for video redundancy at 24fps, would be to dual card record 48fps @360 degrees, then each card would have the equivalent of 24fps @180 degrees.
http://www.captainhook.co.nz/blackmagic-cinema-camera-lut/

**Any post by me prior to Aug 2014 was before i started working for Blackmagic**
Offline

Scott Stacy

  • Posts: 926
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: Lossless CinemaDNG RAW And Compression

PostThu May 07, 2015 1:37 am

CaptainHook wrote:
Scott Stacy wrote:Am I correct in deducing that dual cards are need for 3:1 raw at 60fps on the mini?

It will depend on your card speed, and resolution. 60fps RAW 4:1 @4.6K should be fine to a single card, which is what i shot on the big URSA 4.6K for the demo footage on our site.

Scott Stacy wrote:How is that data reintegrated?

Each alternating frame is recorded to each card and numbered accordingly. Odd numbered frames on one card, even on the other. Because they're sequentially numbered, you can copy both cards into the same folder on your computer and it will be as if they were recorded to a single card from then on, especially as far as software like Resolve is concerned. The audio file is recorded to both cards, for redundancy.

A "trick" for video redundancy at 24fps, would be to dual card record 48fps @360 degrees, then each card would have the equivalent of 24fps @180 degrees.


Thanks, Hook. Appreciate the fast response. Interesting "trick." I was planning on buying Lexar 2.0 Cfast cards but will probably wait and see what develops. Hopefully, a new faster and larger card will emerge to provide some extra headroom.
Scott Stacy, C.S.I.
scott@hppictures.com
Windows 10 version 1803 Build 11/12/18
i7 6950x
2-Titan X
Decklink Pro 4K
Areca 1883 48TB RAID6
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostThu May 07, 2015 5:46 am

Scott, it could be a long wait for faster cards than the larger Lexar CFast 2 cards. It could happen this summer or next summer or never. I posted elsewhere that the write speed of recording media is the current 'wall' you will hit well before the URSA won't be able to handle faster sensors because their processors aren't fast enough. But the wall we hit today applies to ProRes 444 XQ 12bit recording in UHD HFR. It seems very few people consider that a serious limitation. Just trying to say you might want to take advantage of the new lower prices rather than wait for something very much unknown. Or buy two now and two next year, hopefully when they're even faster.


Rick Lang
Sent using Tapatalk HD
Rick Lang
Offline

Scott Stacy

  • Posts: 926
  • Joined: Sun Apr 28, 2013 4:02 pm
  • Location: Kansas City

Re: Lossless CinemaDNG RAW And Compression

PostThu May 07, 2015 1:24 pm

rick.lang wrote:Just trying to say you might want to take advantage of the new lower prices rather than wait for something very much unknown. Or buy two now and two next year, hopefully when they're even faster.

Thanks for the advice, Rick!
Scott Stacy, C.S.I.
scott@hppictures.com
Windows 10 version 1803 Build 11/12/18
i7 6950x
2-Titan X
Decklink Pro 4K
Areca 1883 48TB RAID6
Offline
User avatar

Daniel Bănică

  • Posts: 123
  • Joined: Thu Oct 13, 2016 10:56 am
  • Location: Poland

Re: Lossless CinemaDNG RAW And Compression

PostSat Apr 14, 2018 8:43 pm

Hi @,

So from the requirements point of view on the computer during grading is there any difference between RAW and 3:1 RAW?

Regards!
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 1:27 pm

You need more CPU. GPU needs are the same.
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 1:32 pm

CaptainHook wrote:You're looking at around 310MB/s for Lossless RAW 4608x2592 @24fps.



This is about 1.7:1, which sounds realistic.
Problem with lossless is that you can't guarantee compression ratio. Some frames will compress 4:1, others barely 1.5:1, so when you design encoding/storage you basically still have to think that there may be cases when frame is almost as big a uncompressed. This is one of the lossless mode biggest "problems".
I would prefer hybrid codec which tries to stay at 3:1, but for "easy" frames it switches to lossless compression.
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 4:44 pm

Andrew, when I compared lossless raw CinemaDNG compression in 2016 on the URSA Mini 4.6K, complex frames would compress around 1.2x, most frames would compress 1.3-1.5x.


Sent from my iPhone using Tapatalk
Rick Lang
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 5:28 pm

This means DNG lossless algorithm is very simple, which also points to fact that it should decompress very fast.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:25 pm, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 7:59 pm

Scrap my old post. I was trying to do 3:1, but my source was losslessly compressed (not uncompressed), so at the end I was doing more like 5:1 (I took lossless not uncompressed frame size as reference).
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:32 pm, edited 2 times in total.
Offline

Sean van Berlo

  • Posts: 148
  • Joined: Wed Jul 01, 2015 6:33 am
  • Location: The Netherlands

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 8:12 pm

Thanks for doing that test! That would mean that Prores 422 HQ would deliver better quality than 3:1 Raw Compression? I find that hard to believe, that's crazy!
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 8:15 pm

Bad info here- please ignore.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:21 pm, edited 1 time in total.
Offline
User avatar

rick.lang

  • Posts: 9555
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: Lossless CinemaDNG RAW And Compression

PostSun Apr 15, 2018 10:53 pm

Remember, comparing quality of CinemaDNG to ProRes is not the same as comparing datarates. The datarates are easily compared as they are quantified. Quality is much more subjective. If the datarates are very similar, I would think the quality of the image derived from CinemaDNG will be superior to ProRes. I’d need to do a visual check on CDNG 4:1 versus ProRes HQ to verify my assumption as the datarates for those codecs are very close.


Sent from my iPhone using Tapatalk
Rick Lang
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 12:02 am

Bad info here, please ignore.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:24 pm, edited 1 time in total.
Offline

Mihail Moskov

  • Posts: 37
  • Joined: Wed Jun 24, 2015 11:18 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 9:17 am

Regarding lossless DNG, the compression achieved depends not only on image complexity, but also on the choice of a tonal compression curve. BM cameras do suffer a bit from the lack of OLPFs, but also the tonal curve is not friendly to (lossless) compression down the pipe. Raw from most cameras will compress to around 2:1 using lossless (including most linear raw cameras, as well as Arri log raw), but not BM cameras (around 1.6:1 is typical for these; a bit less in-camera). Actually, putting aside the forementioned impediments, BM lossless in-camera is quite efficient -- slimRAW will typically gain 10 to 20% doing lossless recompression with losslessly compressed raw from other cameras, but will typically only gain up to ~5% on a BM lossless source.

Regarding lossy, there aren't any fundamental differences between DNG or ProRes or DNx, or anything else based on DCT. Now, there are many tricks you can do, and choices to make even within a single codec, which would account, for example, for the differences between DNG 3:1 in-camera and slimRAW DNG 3:1. But fundamentally, these all work similarly. Obviously, I've done hundreds of tests similar to yours (mostly using DNxHR, cause I am on Windows), and you'd need to step up to a 4:4:4 DNx codec version to get to PSNR similar to 3:1 DNG. Actually, the numbers I usually get with 4:2:2 versions are similar, except the difference is in the opposite direction. Of course, it is possible that ProRes is better than DNx. Now, whether PSNR is a good measure in this case is another story. You can google an excellent article by Wang and Bovik on the applicability of MSE based measures (like PSNR). FWIW, DNG 3:1 will consistently have an SSIM score in excess of 0.99, which is excellent structural fidelity (not that SSIM should be trusted blindly, either).

The big difference actually comes from the order of operations: it is debayer -> compress in the intermediate codec case, and compress -> debayer in the raw compression case. In the first case, you interpolate missing samples, and then discard information, in the other case you discard information, then interpolate missing samples. Also, a small tip: dialing down sharpness in the raw tab to 0 will get you better results with lossy raw (especially at higher raw compression ratios).

Regarding speed, there is a correlation between decoding speed and size. Assuming the same algorithm, smaller files decompress faster than bigger files, not the other way around. This is because entropy decoding dominates decoding performance. The performance advantage of a codec like ProRes Raw is that its sliced nature allows for decompression of smaller versions of the data: for example, you can get a half res version of the image without decoding all of it. If you are decoding full res, it won't matter.
www.shutterangle.com
The science & magic of shooting moving pictures

www.slimraw.com
A fast CinemaDNG compressor
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 10:21 am

Bad info here also.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:22 pm, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 10:42 am

Mihail Moskov wrote:Regarding speed, there is a correlation between decoding speed and size. Assuming the same algorithm, smaller files decompress faster than bigger files, not the other way around. This is because entropy decoding dominates decoding performance. The performance advantage of a codec like ProRes Raw is that its sliced nature allows for decompression of smaller versions of the data: for example, you can get a half res version of the image without decoding all of it. If you are decoding full res, it won't matter.


ProRes is the same DCT based codec as DNxHR etc. You can't easily decode half resolution (although Apple SDK allows for it, but no one except Apple ever implemented it). All threading in ProRes is done using frame portion or mainly decoding many frames simultaneously.

Cineform is wavelet based and allows for fraction resolution easy decoding, which is a great feature.
Offline

Mihail Moskov

  • Posts: 37
  • Joined: Wed Jun 24, 2015 11:18 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 11:30 am

Yeah, well, your v210 step might be responsible for what you observe. It is better to skip intermediate chroma subsampling and evaluate only the effects of raw compression vs prores compression. Intermediate subsampling is introducing unnecessary ambiguity. I suggest using 16-bit TIFF or a similar full color high precision format as an intermediate step when doing tests like these; you'll get more meaningful results.

Artifact propagation will depend on the debayer algorithm, there are compression aware debayer algorithms, but yes -- this is the main reason utilizing very high raw compression is generally unreasonable. Note how ProRes Raw is also limited to around 4:1 (~3:1 in the Raw HQ case). The effect of changing sharpness should provide an intuition -- you get to sharpen "fake" detail in both cases, but more so on lossy raw.

You can get PSNR to various values in a bayer array comparison by tweaking quantization as well as channel separation; whether you'll hit 60DB or not will mostly depend on image complexity. The problem with PSNR is that it isn't a perceptual measure and DCT is fundamentally a perceptual compression technique. Basically, what this means is that it doesn't make sense to rely on it too much when tuning parameters (and it is the first thing you try -- automating parameter selection based on optimizing measures).

re: decoding.
With ProRes you can skip to the next slice as soon as you have enough frequencies decoded. This is effectively the same resolution independence as wavelets. Whether a decoder will take advantage of this is a matter of implementation, but it is trivial to implement.
www.shutterangle.com
The science & magic of shooting moving pictures

www.slimraw.com
A fast CinemaDNG compressor
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 12:08 pm

And another bad info.


Scrap all below!

OK- more precise test.
Same Ursa 4.6RAW DNG sequence.
This time I converted 1 uncompressed RAW frame to 3:1 RAW with slimRAW.
Then converted both from RAW DNG into 16bit RAW TIFF (using dcraw). Now I have raw frames which are more viewing friendly (they are simple TIFF files but holding RAW data).
Then I converted original TIFF with RAW data to ProRes and compared all of them in Resolve by eye. Same result. 3:1 RAW shows easily visible differences against original, where ProRes still looks miles better- can't see almost any difference.
Just for a sake of it I measured PSNR and got again about the same 5-6dB difference.

This is an exercise which does hint how well ProRes (as we know it) behaves on bayer RAW data. From what I see it definitely outperforms compressed DNG by quite a margin.
3:1 DNG changes pixels quite a lot (3:1 zoom is enough to see it)- this is already for me enough info as I know that ProRes at 3:1 ratio will keep them so close to the source that you need 6x zoom to start seeing any changes.
If you say that slimRAW is at least as good as BM internal lossy encoding then as I already said- I'm not impressed with 3:1 DNG RAW at all.

As usual I also tried lossless DNG to prove that there are no issues with tools/method.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:46 pm, edited 3 times in total.
Offline
User avatar

Daniel Bănică

  • Posts: 123
  • Joined: Thu Oct 13, 2016 10:56 am
  • Location: Poland

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 1:02 pm

Andrew Kolakowski wrote:OK- more precise test.
Same Ursa 4.6RAW DNG sequence.
This time I converted 1 uncompressed RAW frame to 3:1 RAW with slimRAW.
Then converted both from RAW DNG into 16bit RAW TIFF (using dcraw). Now I have raw frames which are more viewing friendly (they are simple TIFF files but holding RAW data).
Then I converted original TIFF with RAW data to ProRes and compared all of them in Resolve by eye. Same result. 3:1 RAW shows easily visible differences against original, where ProRes still looks miles better- can't see almost any difference.
Just for a sake of it I measured PSNR and got again about the same 5-6dB difference.

This is an exercise which does hint how well ProRes (as we know it) behaves on bayer RAW data. From what I see it definitely outperforms compressed DNG by quite a margin.
3:1 DNG changes pixels quite a lot (3:1 zoom is enough to see it)- this is already for me enough info as I know that ProRes at 3:1 ratio will keep them so close to the source that you need 6x zoom to start seeing any changes.
If you say that slimRAW is at least as good as BM internal lossy encoding then as I already said- I'm not impressed with 3:1 DNG RAW at all.

As usual I also tried lossless DNG to prove that there are no issues with tools/method.

So you're saying that it's better ro record ProRes than 3:1 RAW?
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 1:35 pm

Nope, you can't easily form such a recommendation.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:23 pm, edited 1 time in total.
Offline

Mihail Moskov

  • Posts: 37
  • Joined: Wed Jun 24, 2015 11:18 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 1:51 pm

Why not export TIFFs directly from Resolve? What's the need for dcraw here?

baseline = lossless raw -> tiff
3:1 raw -> tiff
lossless raw -> tiff -> prores -> tiff

These three should be appropriate for calculations.
www.shutterangle.com
The science & magic of shooting moving pictures

www.slimraw.com
A fast CinemaDNG compressor
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 2:08 pm

Because my TIFFs hold RAW data- not final image/debayered one.
I want to see how codecs behave with Bayer pattern monochromatic data.
This way I measured DNG (and standard ProRes) capabilities of RAW encoding before any debayering etc happens.
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 4:59 pm, edited 1 time in total.
Offline

Mihail Moskov

  • Posts: 37
  • Joined: Wed Jun 24, 2015 11:18 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 2:17 pm

Wait, are you saying that dcraw now imports 3:1 raw?
www.shutterangle.com
The science & magic of shooting moving pictures

www.slimraw.com
A fast CinemaDNG compressor
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 2:23 pm

No- it shows them as stripes, so it doesn't support them properly.
It does read lossless though, so in my case I go slimRAW uncompressed->3:1->lossless and then dcraw to TIFF with -D option for keeping RAW as is. This way I have BM RAW data stored in 16bit TIFF and I can do different things with it- like test how different codecs behave with it (and compare to DNG lossy).
Offline

Andrew Kolakowski

  • Posts: 4348
  • Joined: Tue Sep 11, 2012 10:20 am

Re: Lossless CinemaDNG RAW And Compression

PostMon Apr 16, 2018 11:31 pm

Ok, so good that I did not give up so easily. I was to disappointed with DNG lossy quality, so I decided to test it again.
This time I quickly realised that my original DNG source is not uncompressed but lossless, so it means my previous math and exercise trying to hit certain compressed frame size were simply wrong by almost 2x. It means I was in reality testing something like 5:1 mode, not 3:1.

3:1 mode (gave so looked for) 58.5dB PSNR which is more what was expected.

Please ignore my previous crap about DNG lossy being so inefficient. It may not be crazy efficient, but it's not that bad either.

I hope I cleared all crap, if not please let me know and I will clean it.

Return to Cinematography

Who is online

Users browsing this forum: aglyons, Chad Wick, Google [Bot], John Richard, Rodrigo Pereira and 14 guests