Visually lossless CinemaDNG codec?

Hi,

Just curious, is the visually lossless compression employed on the 4K BMCC based on the same compression standard specified by Adobe in the DNG 1.4.0.0 spec (i.e., Lossy JPEG)? If so, does that mean that the image data is essentially being encoded at only 8-bits, or has there been some extension allowing for greater than 8-bit JPEG compression?

Secondly, as I understand the spec, the Adobe DNG JPEG compressed format requires the Bayer data to be demosaiced before compression... If that is the case, does that imply that demosacing is taking place in-camera, and by using this format we won't have the capability to take advantage of more advanced demosaicing algorithms in an offline setting? Or has the on-board compute capability of the camera gotten to the point where offline demosaicing offers negligible gains?

Thanks,

Jason

Hi,

Just curious, is the visually lossless compression employed on the 4K BMCC based on the same compression standard specified by Adobe in the DNG 1.4.0.0 spec (i.e., Lossy JPEG)? If so, does that mean that the image data is essentially being encoded at only 8-bits, or has there been some extension allowing for greater than 8-bit JPEG compression?

Secondly, as I understand the spec, the Adobe DNG JPEG compressed format requires the Bayer data to be demosaiced before compression... If that is the case, does that imply that demosacing is taking place in-camera, and by using this format we won't have the capability to take advantage of more advanced demosaicing algorithms in an offline setting? Or has the on-board compute capability of the camera gotten to the point where offline demosaicing offers negligible gains?

Thanks,

Jason

lossy and lossless are different. and from what john said on his blog its about a 1.5-1.2:1 compression which at UltraHD would be like 8MB/frame or 1080p at 2MB/frame

Hi,

Just curious, is the visually lossless compression employed on the 4K BMCC based on the same compression standard specified by Adobe in the DNG 1.4.0.0 spec (i.e., Lossy JPEG)?

From Adobe's developer info, CinemaDNG supports lossless compression at pretty much any bit depth you feel like throwing at it.

I don't understand why everyone always assumes that all compression is lossy, even though if that were the case you wouldn't be able to download, unpack, install, and hope to actually RUN any software.

I don't understand why everyone always assumes that all compression is lossy, even though if that were the case you wouldn't be able to download, unpack, install, and hope to actually RUN any software.

Absolutely. It is either, uncompressed or compressed using lossless or lossy compression.

However, which one is "visually lossless" compression ?

From Adobe's developer info, CinemaDNG supports lossless compression at pretty much any bit depth you feel like throwing at it.

Absolutely, but you didn't look much in it, did you ? If you were a developing a software that reads CinemaDNG, you would dig deeper. You would have found that CinemaDNG is a image sequence where each video frame is independently encoded using the DNG (Digital Negative) image format. The video stream is stored either as a picture track in an MXF (Material Exchange Format) file or as a sequence of individual frame files.

You would also found that the DNG image file must be one of the following :
- Uncompressed
- JPEG compressed data, either baseline DCT JPEG, or lossless JPEG compression
- Deflate compression (ZIP)(Lossless compression)
- Lossy JPEG

Lossy compression is JPEG only.
JPEG compression is available only in 8bits.

I'm asking again, is "visually lossless" compression lossy or lossless ?

Absolutely. It is either, uncompressed or compressed using lossless or lossy compression.

However, which one is "visually lossless" compression ?

Lossless is lossless. Period.

Absolutely, but you didn't look much in it, did you ?

The thing is that I actually understand the terminology, so I'm not going to waste my time spewing it mindlessly.

If you were a developing a software that reads CinemaDNG, you would dig deeper. You would have found that CinemaDNG is a image sequence where each video frame is independently encoded using the DNG (Digital Negative) image format. The video stream is stored either as a picture track in an MXF (Material Exchange Format) file or as a sequence of individual frame files.

Yes, it's spelled out pretty clearly on the web site, and it's orthogonal to the question you asked.

You would also found that the DNG image file must be one of the following :
- Uncompressed
- JPEG compressed data, either baseline DCT JPEG, or lossless JPEG compression
- Deflate compression (ZIP)(Lossless compression)
- Lossy JPEG

Lossy compression is JPEG only.
JPEG compression is available only in 8bits.

None of that contradicts anything I wrote. It's true that JPEG is an 8-bit codec, but the list you cite clearly states that CinemaDNG also supports lossless compression.

I'm asking again, is "visually lossless" compression lossy or lossless ?

Lossless is lossless. That's it. If it's not lossless, then obviously it's lossy. What I said is that CinemaDNG support lossless compression at whatever bit rate you prefer, and with all the jargon that you're regurgitating without any apparent understanding you've found nothing to contradict that, and you won't no matter how hard you try.

For a definitive answer about what compression method BMD is using on the Pocket Cinema Camera, we'll have to wait until BMD decides to share that information with us. For all we know they're still working on a compression implementation that meets the constraints of the limited bandwidth available on an SD card and don't even know for sure yet. It's also possible that they're working on a Cineform RAW implementation. I suspect that a lot of people would like that quite a bit, and it obviously works quite well given the image quality of the Black Edition of the Hero 3.

As far as I know, the 8bit lossy JPEG part in the DNG specs refers to proxies generated on the fly and has nothing to do with the actual material, so I think we can rule that one out.

I guess BM applies some sort of Huffyuv ore similar mild and fast compression - but that's just a speculation.

Lossless is lossless. Period.

Yep. So when Apple is saying that ProRes 422 HQ is visually lossless, it actually means that it is a lossless codec, not a lossy one ? http://support.apple.com/kb/HT5151

When on wikipedia it is written : "The lossy compression that produces imperceptible differences may be called visually lossless." Does it actually means a lossless compression ? http://en.wikipedia.org/wiki/Image_compression

When the FADGI define it as : "A form or manner of lossy compression where the data that is lost after the file is compressed and decompressed is not detectable to the eye; the compressed data apprearing identical to the uncompressed data." - it means that they don't know what they are talking about.
http://www.digitizationguidelines.gov/t ... lylossless

Yep. So when Apple is saying that ProRes 422 HQ is visually lossless, it actually means that it is a lossless codec, not a lossy one ? http://support.apple.com/kb/HT5151

Unless you've been living under a rock for the past decade, you shouldn't be taking anything Apple says seriously enough to care about what it means, based on Apple's history of spin-doctoring.

Lossless is lossless. Period.

No, I didn't say that.

Lossless is lossless. Period.

No, I didn't say that.

Sorry. Tried to remove too much quotes inside quotes inside quotes.

you shouldn't be taking anything Apple says seriously enough to care about what it means, based on Apple's history of spin-doctoring.

You're the one saying that "visually lossless" is truly lossless. Why are you attacking Apple for potentially saying the same thing?

If "lossless is lossless", then what is "visually lossless"? It's clearly not mathematically lossless, or they'd say that.

Hope this helps with all the bickering...

"Currently the prototype camera I have records only in ProRes 422 @ 10 bit, just like the BMCC. The plan is to add a NEW compressed RAW implementation of DNG. The spec allows for this be lossless (not lossy) and at 1.5:1 – 1.2-:1."

http://johnbrawley.wordpress.com/2013/0 ... e-orginal/

You're the one saying that "visually lossless" is truly lossless. Why are you attacking Apple for potentially saying the same thing?

Where? Not once have I used the asinine phrase "visually lossless" in any context other than quoting you. Everyone by now is either a fanatically loyal Apple fan, or knows that Apple has a penchant for frequently drastic spin. That's hardly news.

If "lossless is lossless", then what is "visually lossless"? It's clearly not mathematically lossless, or they'd say that.

Lossless IS lossless. Obviously, "visually lossless" is marketing fluff. I pointed out that CinemaDNG supports lossless compression, which was listed in the features that YOU listed, yet you seemed to think that it was contradicting that statement, even though in reality it was confirmation that CinemaDNG does, in fact, support lossless compression.

Lossless IS lossless. Obviously, "visually lossless" is marketing fluff. I pointed out that CinemaDNG supports lossless compression, which was listed in the features that YOU listed, yet you seemed to think that it was contradicting that statement, even though in reality it was confirmation that CinemaDNG does, in fact, support lossless compression.

It's not obvious at all. Marketing fluff? If anything it's a marketing mistake, but i doubt it. I think they choose the word very carefully.

So what you're telling me, is that the 4K camera will be able to write at least 350MB per second(if we assume a 1.5:1 compression), and i'm telling you, that you are wrong.

No it can't be ! Visually lossless is not lossless ?!

John Brawley confirmed that the BMPC 4K is lossy DNG compression. https://twitter.com/brawlster/status/321621226265141248

Good news, however, it should not be that 8bit Jpeg lossy compression like in the current DNG specs, it should be 12bits. So Adobe should publish soon a new DNG specs including some new lossy compression. Perfect for all those terabytes of DNG and CinemaDNG that I need to archive !!!


Unless he is like most of us, with "all the jargon that we're regurgitating without any apparent understanding."

So it's lossy then? I don't understand how something that's compressed AND lossy can be considered RAW?

So it's lossy then? I don't understand how something that's compressed AND lossy can be considered RAW?

Putting it all together. It isn't really RAW if it's lossy, and i hope they don't call it that. However, CinemaDNG does support lossless compression, but not at ratios that would make 4K practical.

RED does the same thing. Redcode RAW is claimed to be "virtually lossless", but it is indeed a lossy codec.

It's not obvious at all. Marketing fluff? If anything it's a marketing mistake, but i doubt it. I think they choose the word very carefully.

It's fluff. They're basically trying to avoid admitting that their compression is lossy because of the perception that lossy compression is bad.

So what you're telling me, is that the 4K camera will be able to write at least 350MB per second(if we assume a 1.5:1 compression), and i'm telling you, that you are wrong.

If you think that's what I'm telling you, then you need to work on your reading comprehension skills.

Would it be possible to capture uncompressed (or truly lossless) material in lower resolutions though? I think it would be nice to have that option.

[quote="Tamerlin"]Where? Not once have I used the asinine phrase "visually lossless" in any context other than quoting you.[quote]

Wrong, because you never quoted me. You're obviously confusing messages in the thread. You keep repeating the same meaningless refrain and then getting huffy when you're called out on it. Someone said "visually lossless", to which you said, "Lossless is lossless." That's how you said "visually lossless" is lossless. Scroll up and read your own message; it's right there for all to see. It doesn't matter how YOU try to spin it. Take some time and learn to express yourself properly if you're getting frustrated.

I never made any statement about CinemaDNG not supporting lossless compression. Must've been someone else.

So it's lossy then? I don't understand how something that's compressed AND lossy can be considered RAW?

Whether something is compressed or lossy is not a defining point of RAW, RAW means its sensor data rather than a debayered and processed video or picture file.

Merely compressing raw data doesn't burn in white balance and other settings.

If it is 12bit log lossy compression DNG, I would not panick.

You still have almost every advantage that a lossless DNG have. As for quality - really ? You think you are really gonna lose something at 4k resolution ? Even for greenscreen ? Between 4k 12bit lossy compression DNG (HQ) and lossless 2.5k I would go anyday to 4k.

Whether something is compressed or lossy is not a defining point of RAW, RAW means its sensor data rather than a debayered and processed video or picture file.

In fact, in lossy compression - you need to demosaic it than compress it - however nothing will be burned in - no white balance, sharpen, blur, etc... You would not want to demosaic a lossy compressed bayer pattern - that would give a lot of damage.

If it is 12bit log lossy compression DNG, I would not panick.

You still have almost every advantage that a lossless DNG have. As for quality - really ? You think you are really gonna lose something at 4k resolution ? Even for greenscreen ? Between 4k 12bit lossy compression DNG (HQ) and lossless 2.5k I would go anyday to 4k.

As for the quality, we don't know yet. If we have to make an educated guess, I think we'll need a compression ration of at least 3to1. Which means we will lose some flexibility(however minor) in post. Just look at the 1D-C, there is no way to downscale the 4K to get a 1080 image that is as flexible as what you get from the BMCC

If you want, you should try http://www.adobe.com/support/downloads/product.jsp?product=106&platform=Windows to convert some lossless DNG to lossy DNG and you'll see to you have the exact same latitude than RAW - full white balance, highlight and shadow recovery etc. In fact, you should try a DNG that you already manipulated and you'll see that it is at 99.9% the same thing (Some people/company like to call it "visually lossless"). The only problem I got was on extreme cc curves, or extreme recovery I was getting some banding because the current specs, the image data is encoded as 8bit Jpeg. But as a 12 bit ? I trust that.

Lossy DNG to MPG, AVCHD, etc, with burned white balance, burned dynamic range (highlight/shadow already clipped to some degree to maximize the lo bit-depth) burned sharpening, etc..... there is no comparison.

As for the quality, we don't know yet. If we have to make an educated guess, I think we'll need a compression ration of at least 3to1. Which means we will lose some flexibility(however minor) in post. Just look at the 1D-C, there is no way to downscale the 4K to get a 1080 image that is as flexible as what you get from the BMCC

Whether something is compressed or lossy is not a defining point of RAW, RAW means its sensor data rather than a debayered and processed video or picture file.

In fact, in lossy compression - you need to demosaic it than compress it - however nothing will be burned in - no white balance, sharpen, blur, etc... You would not want to demosaic a lossy compressed bayer pattern - that would give a lot of damage.

At which point its not really RAW. Once you demosaic you've changed it from being sensor data into a picture. If you do ANYTHING other than compress the sensor data it's not raw.

Fairly sure .r3d is a lossy system and yet is compressed raw that still requires a debayer.

At which point its not really RAW. Once you demosaic you've changed it from being sensor data into a picture. If you do ANYTHING other than compress the sensor data it's not raw.

Fairly sure .r3d is a lossy system and yet is compressed raw that still requires a debayer.

Your right, it seems there is some methods to do lossy compression on bayer data.
http://www.eurasip.org/Proceedings/Eusipco/Eusipco2006/papers/1568981744.pdf
http://etrij.etri.re.kr/Cyber/Download/PublishedPaper/2802/28-02-21.pdf

By the way, just found that interesting forum thread about the current DNG lossy 8bit jpeg compression. Eric Chan (MadMadChan2000) is one of the DNG designer.
http://forums.adobe.com/message/4137184#4137184

No it can't be ! Visually lossless is not lossless ?!

John Brawley confirmed that the BMPC 4K is lossy DNG compression. https://twitter.com/brawlster/status/321621226265141248

Good news, however, it should not be that 8bit Jpeg lossy compression like in the current DNG specs, it should be 12bits. So Adobe should publish soon a new DNG specs including some new lossy compression. Perfect for all those terabytes of DNG and CinemaDNG that I need to archive !!!


Unless he is like most of us, with "all the jargon that we're regurgitating without any apparent understanding."

Tim, from BMD has stated in an interview that the BMPC4K is using a compression that is akin to a ZIP data compression, not an image compression, and that you will not be able to see any difference than the uncompressed video on the BMCC. That means it is lossless. He used the terms mathematically and visually lossless. Who knows who is right (maybe at this time they both are right because they have different information) but we shall see when the camera ships in a few months hopefully.

Rick Lang
Sent from my iPad using Tapatalk HD

Mathematically lossless compression would be pretty awesome. They will have to pull this off in order for the pocket camera to reduce the bandwidth enough for current SDXC card specs. If they cant pull it off then the pocket camera indeed will not be able to support "raw".

If they have the ability to do this, I am wondering why it hasn't been implemented on the current BMCC in order to reduce the bandwidth to the SSD.

Edit: I'm talking about the Pocket Cinema Camera here.

I think it's important to point out that all the verbage on their site says "lossless CinemaDNG RAW" and not visually lossless.

Screenshots for the record:

Blackmagic Pocket Cinema Camera features a built in SD card recorder that captures [...] lossless compressed CinemaDNG files to fast SDXC cards.

Seems pretty clear cut that they are claiming it can record mathematically lossless "RAW" files to existing SD cards. If it's mathematically lossless, I see no reason why they couldn't make it true RAW - there is no need to debayer in-camera.

I think it's important to point out that all the verbage on their site says "lossless CinemaDNG RAW" and not visually lossless.

Hi Erik. Everything you are refering to is the Pocket Camera. This one is lossless.

The "visually lossless" is the Production 4k camera.

I think it's important to point out that all the verbage on their site says "lossless CinemaDNG RAW" and not visually lossless.

Hi Erik. Everything you are refering to is the Pocket Camera. This one is lossless.

The "visually lossless" is the Production 4k camera.

Thanks for pointing that out.

Seems pretty clear cut that they are claiming it can record mathematically lossless "RAW" files to existing SD cards. If it's mathematically lossless, I see no reason why they couldn't make it true RAW - there is no need to debayer in-camera.

They've gone out of their way to tout that. I eagerly await "the beef"....
That is an obscure Wendy's reference you youngins and international guys might not get....