BRAW? So 3:1 compression is RAW????

The place for questions about shooting with Blackmagic Cameras.
  • Author
  • Message
Offline

Spencer Acoustics

  • Posts: 65
  • Joined: Sat Sep 01, 2018 6:40 pm
  • Location: Dallas, Texas
  • Real Name: Randy Spencer

BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 5:21 pm

I was a bit surprised to find out that BRAW is not really a RAW format, it is a 3:1 compressed format at the highest quality settings. I thought RAW was, well, RAW. As in ALL the data in a uncompressed format. Unless BM adds a true uncompressed BRAW option, in my opinion, this will slow down BM's efforts to make this a real powerhouse of a codec.

Am I wrong here? I LOVE the new codec, I just feel a bit mislead about it being RAW. Yes you get the post production benefits of RAW but is that the only reason BM calls it RAW?
Spencer Acoustics

P4K, Rokinon Cine DS lenses
Asus Deluxe x299 motherboard
Intel 7940X 14 core cpu
32GB ram
Samsung 970 EVO M.2 for programs
Samsung 960 EVO ssd for working files
4-4TB WD Blue for storage
EVGA FTW 1050TI 4GB
Windows 10
DR 16 Studio beta
Offline

Vanni Mastrantonio

  • Posts: 66
  • Joined: Tue May 24, 2016 7:46 pm

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 5:24 pm

Also R3D codec is RAW, but RED never release the 1:1 unconpressed


Inviato dal mio iPhone utilizzando Tapatalk
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 6:04 pm

Spencer Acoustics wrote:I was a bit surprised to find out that BRAW is not really a RAW format, it is a 3:1 compressed format at the highest quality settings. I thought RAW was, well, RAW. As in ALL the data in a uncompressed format. Unless BM adds a true uncompressed BRAW option, in my opinion, this will slow down BM's efforts to make this a real powerhouse of a codec.

Am I wrong here? I LOVE the new codec, I just feel a bit mislead about it being RAW. Yes you get the post production benefits of RAW but is that the only reason BM calls it RAW?


BRAW was explained during the introduction of BRAW at IBC in 2018 september 14 https://www.blackmagicdesign.com/media/release/20180914-01

And explained in this video



And it was published on every website that has anything to do with moving pictures.

And by the way, planet earth is not flat...
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 6:15 pm

Spencer Acoustics wrote:I was a bit surprised to find out that BRAW is not really a RAW format, it is a 3:1 compressed format at the highest quality settings. I thought RAW was, well, RAW.


Are you basing the assertion that it's not raw on the fact that it's not uncompressed?

Are you aware of the fact that even most of BMD's cDNG options were compressed, and most of the compression ratios were based on lossy compression?

None of that is in any way related to whether or not the codec is a raw codec...
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline
User avatar

Peter Benson

  • Posts: 344
  • Joined: Thu Dec 21, 2017 5:12 pm
  • Location: Eastern Time Zone, USA

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 8:33 pm

Spencer Acoustics wrote:I was a bit surprised to find out that BRAW is not really a RAW format, it is a 3:1 compressed format at the highest quality settings.

...I LOVE the new codec, I just feel a bit [misled] about it being RAW. Yes you get the post production benefits of RAW but is that the only reason BM calls it RAW?
Could a possible additional reason relate to competitors naming their mildly compressed formats similarly (e.g., Apple ProRes Raw)?
DTV 10.9.7 > Kingston SD5000T > MiniMonitor > Bravia | Samsung U28D590 | DRS 14.3.0.014 | Win8.1 x64 | ASUS G751JL, i7-4720HQ, 24GB | GTX965M | 1TB HDD, 500GB EVO 850 SSD | MCU Pro | Softube Console 1 Mkii | Shuttle Pro 2
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 8:46 pm

It's probably closer to RAW than to "normal" format.
I'm still waiting so see if you can get RAW Bayer data through SDK. If you can't then it's definitely not "proper" RAW format. At the end it doesn't mean that much, but facts should be kept straight.
Offline

Spencer Acoustics

  • Posts: 65
  • Joined: Sat Sep 01, 2018 6:40 pm
  • Location: Dallas, Texas
  • Real Name: Randy Spencer

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 11:07 pm

BRAW was explained during the introduction of BRAW at IBC in 2018 september 14 https://www.blackmagicdesign.com/media/ ... 0180914-01

And explained in this video


I did not see the press release but I did watch the video just hours after it was posted. That was a few months ago so I don't remember the details, only that I was excited that BRAW would be part of my new P4K.

Are you basing the assertion that it's not raw on the fact that it's not uncompressed?


I am new to this level of video equipment/photography equipment so my understanding of industry standards and lingo is definitely lacking. Based on everything I've read to date, RAW is an uncompressed format. Now ya'll are saying that's not true. OK, I'm a newbie, that's why I asked the question.

RAW really means it's just data that is interpreted into an image either in the camera or in post-processing


OK, now I understand. But why start at 3:1 compression? Shouldn't they offer a "lossless" format as well?

Granted, I am thoroughly impressed with all of the test footage I have shot over the last two days. I believe I am getting much cleaner looking video with BRAW 5:1 AND 8:1 than I ever did with DNG 4:1. Of course, I'm also getting a much better understanding of how to properly use the camera, cine lenses, etc. This is all just part of my learning curve boys!
Spencer Acoustics

P4K, Rokinon Cine DS lenses
Asus Deluxe x299 motherboard
Intel 7940X 14 core cpu
32GB ram
Samsung 970 EVO M.2 for programs
Samsung 960 EVO ssd for working files
4-4TB WD Blue for storage
EVGA FTW 1050TI 4GB
Windows 10
DR 16 Studio beta
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostThu Mar 07, 2019 11:55 pm

Q0 goes upto 2:1 compression, that is (almost) lossless.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 12:02 am

Australian Image wrote:
Spencer Acoustics wrote:Am I wrong here? I LOVE the new codec, I just feel a bit mislead about it being RAW. Yes you get the post production benefits of RAW but is that the only reason BM calls it RAW?


It's still RAW even though it's compressed. This has been going on in the photography world for since RAW was conceived for digital cameras. Digital camera have been providing compressed and uncompressed RAW files for some time. Some even claim uncompressed, but do some compression anyway.

RAW really means it's just data that is interpreted into an image either in the camera or in post-processing, otherwise it's meaningless. Also look at compressed RAW somewhat like a zip file in data terms; the file contains all the data, only the empty spaces have been removed.


As you said it's not about compression. RAW format is the one where you can get RAW Bayer data out of it. If BM RAW doesn't provide it (as it partially debayers in camera ) it means it's not strictly RAW format.
Offline

Aaron Hinton

  • Posts: 57
  • Joined: Sat Mar 07, 2015 5:46 pm

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 12:19 am

Trust me, you don't actually want uncompressed RAW. It's a pain in the butt to work with and takes up so much storage that you'll be constantly purchasing new drives. BRAW is intended to have all the benefits of RAW, but without all that headache. It's essentially Blackmagic's answer to REDCODE, and it's fantastic.

RED doesn't even put uncompressed RAW on their cameras, despite people constantly asking for it, because they understand how much of a pain it would really be, and how little it would actually benefit the user.
Aaron Hinton
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 12:24 am

Aaron Hinton wrote:RED doesn't even put uncompressed RAW on their cameras, despite people constantly asking for it, because they understand how much of a pain it would really be, and how little it would actually benefit the user.


To go even further, Red says that 2:1 compression is mathematically lossless. The VFX pros recommend 4:1 or 5:1 compression for detailed compositing work, rather than 2:1.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 12:56 am

Rakesh Malik wrote:To go even further, Red says that 2:1 compression is mathematically lossless. The VFX pros recommend 4:1 or 5:1 compression for detailed compositing work, rather than 2:1.


Red uses Wavelet encoding (jpeg2000) and 2:1..3:1 can be mathematically lossless.
With 8k.R3D you can't go lower than 5:1 with 24 fps (that is the best it can do).

With 5k.R3D you can go all the way down to 2:1 with 24 fps (251 MByte/s) and you need it when you want to compete with UMP G2 Q0, X-OCN XT or ARRIRAW.

Lossy compression kills the image.
Offline

Uli Plank

  • Posts: 4767
  • Joined: Fri Feb 08, 2013 2:48 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 2:33 am

Big feature films use 6:1 and even 8:1 for scenes which don't need VFX.
Resolve Studio and Fusion Studio
iMac 2017 Radeon Pro 580 8 GB VRAM and 32 GB RAM
Offline

Uli Plank

  • Posts: 4767
  • Joined: Fri Feb 08, 2013 2:48 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 3:02 am

This is highly dependent on image content. If you stress Q5 with high detail and compare Q0 at very low detail, they touch each other's range. So, we'll probably never see Q1 to Q4.
Resolve Studio and Fusion Studio
iMac 2017 Radeon Pro 580 8 GB VRAM and 32 GB RAM
Online
User avatar

rick.lang

  • Posts: 11092
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 4:39 am

Misha, you said “lossy compression kills the image.” That might be a little harsh. Here’s a brief summary of how raw has evolved on BMD cameras.

Uncompressed:
On the original BMCC, uncompressed gave you all the sensor data in 12bits per photosite (CinemaDNG).

Mathematically Lossless Compressed:
The uncompressed raw was effectively zipped in camera so that it took up less space but could be unpacked without any loss of information in the NLE. The degree of compression depended on the content of the frame and could be between about 1.15-1.4:1. I found 1.25:1 was typical for me but BMD would say about 1.3:1 space savings.

Visually Lossless Compressed (Lossy)
The uncompressed data undergoes lossy data compression. Information is lost in that you cannot get back to uncompressed raw in the NLE. But the compression is mild raw 3:1 and 4:1 so that with moving pictures you are not going to notice anything is missing in almost all situations. When you pixel peep still frames you can see differences in detail but for the most part colour looks accurate. BMD would not compress more than 4:1 as they felt going to higher compression did show information was lost in motion video. The ratios are in terms of uncompressed raw, so 4:1 means the video is a quarter the uncompressed space.

BRAW (Visually Lossless)
Constant Bitrate
Fixed data rates for 3:1, 5:1, 8:1, 12:1 compression and should be visually lossless in most situations with motion video.
Constant Quality
Variable data rates for Q0 and Q5.
Q0 can be as lightly compressed as 2:1 274MB/s compared to uncompressed raw and as heavily compressed as 5:1 uncompressed raw 110MB/s.
Q5 can be a lightly compressed as about 7:1 78MB/s and as heavily compressed as 20:1 27MB/s.
NB the above BRAW bitrates were taken from the chart in Grant’s UBC 2018 video and may have changed.

Sent from my iPhone using Tapatalk
Rick Lang
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 7:15 am

Spencer Acoustics wrote:I was a bit surprised to find out that BRAW is not really a RAW format, it is a 3:1 compressed format at the highest quality settings. I thought RAW was, well, RAW. As in ALL the data in a uncompressed format. Unless BM adds a true uncompressed BRAW option, in my opinion, this will slow down BM's efforts to make this a real powerhouse of a codec.

Am I wrong here? I LOVE the new codec, I just feel a bit mislead about it being RAW. Yes you get the post production benefits of RAW but is that the only reason BM calls it RAW?


It's a bit complex. As you may know, as far as advertising claims go Raw, is actually virtually unprocessed sensir data, wherever 4:4:4, 4:2:2, 4:2:0 Bayer (the raw mistakenly called raw) rgbw, yellow magenta white, etc.

So, as far as it represents virtually untouched sensor data, it is Raw bayer. This may include the thought of outputing at a combined lower resolution, as old caneeas with component oupur did in the early century and before.

But Braw seems to possibly be just normalish video that bayer values can be extracted from and reprocessed. It also seems to be processed removing your ability to do it yourself. I would argue that elimination of noise to authentic pixel value representative of the actual received light, might also be considered raw bayer, but that's my opinion, and may not be considered valid. But how far does this processing go before it is just restorative and not raw?

In a sense, the usage, and legal definition of raw may expand, but as far as I am concerned, the legal definition should stay centered on the true meaning of raw,not s morning t essentially raw with or without processing to enhance its rawness. In this way, you you need to add controls as we have been discussing to turn down processing to a raw level that just produces an authentic raw pixel value. Some people may want to turn that off completely to produce an unsharpened cdng like image.
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 10:33 am

Australian Image wrote:I have to say that I'm amazed at the recording time difference between Constant Quality Q0 and Q5. The former on a 1TB drive is 153 min and the latter is 611 min.


Number suggest you the difference. Maybe not directly but at least roughly so about 5:1 (which is not far from real numbers).
Last edited by Andrew Kolakowski on Fri Mar 08, 2019 10:55 am, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 10:36 am

Rakesh Malik wrote:
Aaron Hinton wrote:RED doesn't even put uncompressed RAW on their cameras, despite people constantly asking for it, because they understand how much of a pain it would really be, and how little it would actually benefit the user.


To go even further, Red says that 2:1 compression is mathematically lossless. The VFX pros recommend 4:1 or 5:1 compression for detailed compositing work, rather than 2:1.



With lossless there is no single number. 2:1 is achievable, but sometimes may go bit down for some scenes (very noisy ones). Good lossless codecs can go up to 3:1 or even 4:1 but on easy/clean footage (eg, CGI), but you can't even guarantee the rate (which is a bit of problem).
Last edited by Andrew Kolakowski on Fri Mar 08, 2019 10:45 am, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 10:44 am

MishaEngel wrote:
Rakesh Malik wrote:To go even further, Red says that 2:1 compression is mathematically lossless. The VFX pros recommend 4:1 or 5:1 compression for detailed compositing work, rather than 2:1.


Red uses Wavelet encoding (jpeg2000) and 2:1..3:1 can be mathematically lossless.
With 8k.R3D you can't go lower than 5:1 with 24 fps (that is the best it can do).

With 5k.R3D you can go all the way down to 2:1 with 24 fps (251 MByte/s) and you need it when you want to compete with UMP G2 Q0, X-OCN XT or ARRIRAW.

Lossy compression kills the image.


You can't guarantee ratio with lossless encoding. There is no chance you can have 3:1 on every source. Even 2:1 is pushing hard, specially for out of sensor (noisy) data.
Codex has some lossless codec made specially for Arri RAW and it's about 2:1. More precisely 1.7:1. Even if they made it specially for RAW data still could not guarantee 2:1.
3:1 lossless one very source/frame is a dream.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 10:48 am

Wayne Steven wrote:
Spencer Acoustics wrote:But Braw seems to possibly be just normalish video that bayer values can be extracted from and reprocessed. It also seems to be processed removing your ability to do it yourself. I would argue that elimination of noise to authentic pixel value representative of the actual received light, might also be considered raw bayer, but that's my opinion, and may not be considered valid. But how far does this processing go before it is just restorative and not raw?


Cineform RAW cmd tool was re-creating RED RAW data from debayered one, as RED never lest you access RAW data in their SDK.
Last edited by Andrew Kolakowski on Fri Mar 08, 2019 10:55 am, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 10:51 am

Australian Image wrote:I just did some more comparisons and Constant Bitrate (CB) 12:1 is also 611 min, with 3:1 154 min, the same as Constant Quality (CQ) Q5 and Q0. The real differences occur when you use CB 5:1 or 8:1. I'm still not quite sure which option would really be the optimum compromise between quality and file size, but I'm swaying toward CB 5:1.

I should add that this is predicated not just on producing video, but also extracting usable stills (I need both).


If storage is not an issue CQ options are better as they follow quality of the particular frame. In such a case quality of the whole source should be more even. With constant modes when you shoot clean scene and then noisy then noisy part will be relatively worse quality. There is also other point- how often do you shot in one recording mixed scenes: low detailed/detailed/noisy/clean etc. There is also a question how well CQ modes are designed and how well respond frames difficulty changes.
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 1:51 pm

Andrew Kolakowski wrote:
MishaEngel wrote:
Rakesh Malik wrote:To go even further, Red says that 2:1 compression is mathematically lossless. The VFX pros recommend 4:1 or 5:1 compression for detailed compositing work, rather than 2:1.


Red uses Wavelet encoding (jpeg2000) and 2:1..3:1 can be mathematically lossless.
With 8k.R3D you can't go lower than 5:1 with 24 fps (that is the best it can do).

With 5k.R3D you can go all the way down to 2:1 with 24 fps (251 MByte/s) and you need it when you want to compete with UMP G2 Q0, X-OCN XT or ARRIRAW.

Lossy compression kills the image.


You can't guarantee ratio with lossless encoding. There is no chance you can have 3:1 on every source. Even 2:1 is pushing hard, specially for out of sensor (noisy) data.
Codex has some lossless codec made specially for Arri RAW and it's about 2:1. More precisely 1.7:1. Even if they made it specially for RAW data still could not guarantee 2:1.
3:1 lossless one very source/frame is a dream.



2:1..3:1 can be mathematically lossless

As far as I know:
ARRIRAW is 16 bits in a 12 bits logged container so it's 16:12 compressed.
R3D is Wavelet compressed (jpeg2000) and 16 bits in a 12 bits logged container.
X-OCN is compressed in a 16 bit lineair container.
BRAW is partially de-bayered and 16 bits in a 12 bits logged container.
CineformRAW is Wavelet compressed and 16 bits in a 12 bits logged container.

When I had free choice I would go for CineformRAW and since I haven't it's BRAW and cDNG.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 2:16 pm

Yes, it can be 10:1 as well for simple CGI content, but this doesn't mean much. In case of camera recording you would like guaranteed rate (so you can plan recoding module, speed, storage, timings etc), but for lossless encoding it's basically impossible. You can just estimate range eg. 1.5:1-3:1. Same happens with BM RAW CQ modes (here at least you can by design specify min and max ratio). What can be done is lossless mode which switches to lossy when ratio goes to low (HDCAM-SR works opposite- it's compressed around 4:1, but when it can it will switch to lossless mode).
Here we are talking about real (quite often noisy) footage out of sensor so it's worse case for compression and guaranteed 3:1 is simply unrealistic.
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 2:24 pm

Andrew Kolakowski wrote:In case of camera recording you would like guaranteed rate (so you can plan recoding module, speed, storage, timings etc), but for lossless encoding it's basically impossible.


When we use our cameras for news gathering that might be true. When we shoot commercials we don't care about storage needs only picture quality is what counts. UMP auto-switches over from one storage unit to the other. We have lot's of 2TB samsung pro drives to hook on.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 2:38 pm

These are different aspects of recording "problems".
You may have 1000 SSDs, but this won't enable for example high frame rates as rate will be too high for lossless encoding. You then have to for example design good caching system (to avoid spikes for some frames) in camera with lots of RAM, which makes price going up etc.
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 3:11 pm

Andrew Kolakowski wrote:
Wayne Steven wrote:
Spencer Acoustics wrote:But Braw seems to possibly be just normalish video that bayer values can be extracted from and reprocessed. It also seems to be processed removing your ability to do it yourself. I would argue that elimination of noise to authentic pixel value representative of the actual received light, might also be considered raw bayer, but that's my opinion, and may not be considered valid. But how far does this processing go before it is just restorative and not raw?


Cineform RAW cmd tool was re-creating RED RAW data from debayered one, as RED never lest you access RAW data in their SDK.


Andrew, mate. I'm having a bit of trouble with this one. What do you mean?
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Offline
User avatar

joe12south

  • Posts: 400
  • Joined: Tue Jan 16, 2018 4:14 pm
  • Location: Nashville, TN
  • Real Name: Joseph Moore

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 3:21 pm

Ultimately, it matters not what's going on under the hood if the quality and throughput meet my needs.

That said...I can't help but be curious to know more about what "partially debayered" actually means. Which steps are being done in-camera (and thus out of influence by the end user) and which are being done in post?

BMD is typically frustratingly secretive, so we may never know. I guess I'll just have to go out and try my best to capture great images. ;-P
Pocketluts: Purpose-built LUT system for the Black Magic Design Pocket Cinema Camera 4K
http://www.pocketluts.com/
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 3:35 pm

joe12south wrote:Ultimately, it matters not what's going on under the hood if the quality and throughput meet my needs.

That said...I can't help but be curious to know more about what "partially debayered" actually means. Which steps are being done in-camera (and thus out of influence by the end user) and which are being done in post?

BMD is typically frustratingly secretive, so we may never know. I guess I'll just have to go out and try my best to capture great images. ;-P


Well R3D, ProResRAW and X-OCN are also encrypted, the only one that's open is CineformRAW and it's the best RAW format available in my opinion: Descent compression/quality ratio, low on computer resources, free to use, opensource.
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 5:19 pm

MishaEngel wrote:R3D is Wavelet compressed (jpeg2000) and 16 bits in a 12 bits logged container.


It's 16-bit linear.

When I had free choice I would go for CineformRAW and since I haven't it's BRAW and cDNG.


It's a shame that hardly anyone picked up Cineform even after the SMPTE made it an open standard.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 6:00 pm

MishaEngel wrote:
joe12south wrote:Ultimately, it matters not what's going on under the hood if the quality and throughput meet my needs.

That said...I can't help but be curious to know more about what "partially debayered" actually means. Which steps are being done in-camera (and thus out of influence by the end user) and which are being done in post?

BMD is typically frustratingly secretive, so we may never know. I guess I'll just have to go out and try my best to capture great images. ;-P


Well R3D, ProResRAW and X-OCN are also encrypted, the only one that's open is CineformRAW and it's the best RAW format available in my opinion: Descent compression/quality ratio, low on computer resources, free to use, opensource.


ProResRAW or X-OCN are not encrypted as far as I know. Only RED uses encryption, as RED doesn't want anyone to be able to decode their files in other way than through their SDK.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 6:05 pm

Rakesh Malik wrote:
MishaEngel wrote:R3D is Wavelet compressed (jpeg2000) and 16 bits in a 12 bits logged container.


It's 16-bit linear.

When I had free choice I would go for CineformRAW and since I haven't it's BRAW and cDNG.


It's a shame that hardly anyone picked up Cineform even after the SMPTE made it an open standard.


SMPTE VC-5 standard is not the same as free public Cineform as far as I know.
Cineform never had code for ASIC and also it's very VBR alike nature was problematic from recording point of view. Today it's less of a problem (we have ProREsRAW, BM RAW CQ modes).

From interview with David:

"2. So far, there’s been only one camera that would record directly to Cineform (to my knowledge). Why not more prosumer or pro cameras? Why is it difficult to persuade camera manufacturers?

David Newman: We are software company with only 8 people, mostly engineers. The camera guys generally need an FPGA or ASIC implementation for lower power usage and long battery life. While the CineForm codec is designed with potential hardware implement in mind, and will have excellent low power characteristic, it will be the responsibility of a licensing partner to do the hardware port. In some ways we’re a victim on our software successes and Intel’s every increasing performance, hardware vendors from Silicon Imaging, Wafian, CineDeck and 1Beyond, all simply use a mobile Intel CPU for live CineForm encoding, which constantly get faster and cheaper — pushing back the need for a hardware port."
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 6:51 pm

Andrew Kolakowski wrote:SMPTE VC-5 standard is not the same as free public Cineform as far as I know.


The Cineform source code wasn't open sourced at the same time, but since the specs were standard, anyone who wanted to could have implemented them.

The reality is that by the time that happened, most of the companies that might have been interested had most likely already embarked down the R&D for their own clip-based codec option. It's not like X-OCN and braw just happened overnight; BMD's been working on braw for years -- more than two according to Grant Petty when he announced it at IBC... so basically it was too late.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 7:08 pm

Rakesh Malik wrote:
MishaEngel wrote:R3D is Wavelet compressed (jpeg2000) and 16 bits in a 12 bits logged container.


It's 16-bit linear.

[quote]

Do the math and you will find out that it's 16 bits logged in a 12 bits container, as far as I know only Sony's X-OCN is 16 bits linear.
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 7:10 pm

David always wanted to make Cineform an SMPTE standard as this makes codec more reputable with clear and available spec. During that process codec was tweaked a bit, but this makes VC-5 not 100% compatible with "public" version.
Online
User avatar

rick.lang

  • Posts: 11092
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 7:11 pm

More than two years in development may explain why BRAW beta appeared on the URSA Mini Pro first, then BMPCC4K, and why it may be more of a challenge to port it to the URSA Mini 4.6K Camera that predates all efforts to create BRAW.


Sent from my iPhone using Tapatalk
Rick Lang
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 7:13 pm

MishaEngel wrote:Do the math and you will find out that it's 16 bits logged in a 12 bits container, as far as I know only Sony's X-OCN is 16 bits linear.


You know not very far then.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 8:08 pm

Wayne Steven wrote:
Andrew Kolakowski wrote:Cineform RAW cmd tool was re-creating RED RAW data from debayered one, as RED never lest you access RAW data in their SDK.


Andrew, mate. I'm having a bit of trouble with this one. What do you mean?


You can't access RAW data with RED SDK so in order to convert RED RAW to CF RAW, David was debayering RED RAW and then interpolating RAW data and converting to CF RAW. Convoluted (+ very slow) and not very good way, but there isn't other one. With Arri, BM etc you can easily access RAW data, so then just encode it to CF RAW, which is very fast.
This is related to fact that you said that RAW can be with some precision "restored" from already debayered image.
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 9:56 pm

Rakesh Malik wrote:
MishaEngel wrote:Do the math and you will find out that it's 16 bits logged in a 12 bits container, as far as I know only Sony's X-OCN is 16 bits linear.


You know not very far then.


From the RED website https://www.red.com/recording-time

DSMC2 MONSTRO
8K FULL FORMAT
24 FPS
REDCODE 5:1
259 MByte/s

(16bits per pixel x 8192 pixels horizontal x 4320 pixels vertical x 24 fps) / 8 bits = 1,698,693,120 Byte/s

5:1 compression 1,698,693,120 Byte per second/5 = 339,738,624 Byte/s

339,738,624 Byte/s x 12/16(16 bits in a logged 12 bits container) = 254,803,968 Byte/s

254,803,968 Byte/s + metadata + error correction ~ 259 MByte/s
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 9:59 pm

Andrew Kolakowski wrote:
Wayne Steven wrote:
Andrew Kolakowski wrote:Cineform RAW cmd tool was re-creating RED RAW data from debayered one, as RED never lest you access RAW data in their SDK.


Andrew, mate. I'm having a bit of trouble with this one. What do you mean?


You can't access RAW data with RED SDK so in order to convert RED RAW to CF RAW, David was debayering RED RAW and then interpolating RAW data and converting to CF RAW. Convoluted (+ very slow) and not very good way, but there isn't other one. With Arri, BM etc you can easily access RAW data, so then just encode it to CF RAW, which is very fast.
This is related to fact that you said that RAW can be with some precision "restored" from already debayered image.


https://topazlabs.com/jpeg-to-raw-ai/
Offline

Andrew Kolakowski

  • Posts: 5578
  • Joined: Tue Sep 11, 2012 10:20 am

Re: BRAW? So 3:1 compression is RAW????

PostFri Mar 08, 2019 11:34 pm

It has nothing to do with RED RAW data. You can't access it as it's encrypted. Only way to process RED RAW is using RED SDK and this will only give you debayered video.
It's like protected DCP- you need a key to watch it.
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 4:00 am

Thanks for that before Andrew.

Any louped images of the Red sensors color filter patterns. Pretty strange stuff the hiding if the, supposedly, bayered data.

Do you know, bayer had come up with a complimentary color filtered version at the same time, but they could make the color filter fine enough with that in those day?

How do we really know what they are hiding down there?
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 4:13 am

Andrew Kolakowski wrote:It's a shame that hardly anyone picked up Cineform even after the SMPTE made it an open standard.


SMPTE VC-5 standard is not the same as free public Cineform as far as I know.
Cineform never had code for ASIC and also it's very VBR alike nature was problematic from recording point of view. Today it's less of a problem (we have ProREsRAW, BM RAW CQ modes).

From interview with David:

"2. So far, there’s been only one camera that would record directly to Cineform (to my knowledge). Why not more prosumer or pro cameras? Why is it difficult to persuade camera manufacturers?

David Newman: We are software company with only 8 people, mostly engineers. The camera guys generally need an FPGA or ASIC implementation for lower power usage and long battery life. While the CineForm codec is designed with potential hardware implement in mind, and will have excellent low power characteristic, it will be the responsibility of a licensing partner to do the hardware port. In some ways we’re a victim on our software successes and Intel’s every increasing performance, hardware vendors from Silicon Imaging, Wafian, CineDeck and 1Beyond, all simply use a mobile Intel CPU for live CineForm encoding, which constantly get faster and cheaper — pushing back the need for a hardware port."


I thought the gopro 6 HD a version of that in hardware. He might say they only have 8 engineers (doing what, a new codec already done) but they were bought by go pro. Of course there was plenty of flexibility to do an asic design core to sell to manufacturers at a profit.

I remember years ago, David debating with me that they hadn't done any fpga design, after he had announced a recorder product project and had it up on the website, and I had read what he had been saying. It suddenly vanished, and they were bought by go pro, after one of the more strange takeovers that s type of thing. A company producing personal action cameras, called go pro, buys a professional video products company, but doesn't go pro by implenting it in their action camera or release a professional camera, but leaves it just as a software by product. Then the standardisation process that didn't go anywhere much.
Last edited by Wayne Steven on Sat Mar 09, 2019 7:16 am, edited 2 times in total.
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Offline

Steven Abrams

  • Posts: 266
  • Joined: Tue Feb 24, 2015 12:43 am
  • Location: LA La Land

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 4:45 am

Rakesh Malik wrote:The reality is that by the time that happened, most of the companies that might have been interested had most likely already embarked down the R&D for their own clip-based codec option.

There's another factor - red did a licensing agreement with cineform. If you try make a camera with cineform raw on it, red will come for you as it clearly violates their patents (patents that reference cineform as prior art).

So while it was made "open", in practice no one can easily use it in camera. Go ahead and try and see what happens. I have friends that were investigating this for a camera they wanted to develop, cineform raw and VC5 turned into a dead end because of licensing and patent issues when adding into hardware.

http://cineform.blogspot.com/2017/10/ci ... 6256120171

David Newman wrote:For 4:2:2 and 4:4:4 in camera encoding we are completely patent free, so hardware vendors are welcome to build CineForm compliant encoding and decoding of all formats except for RAW. The direct using of the CineForm-SDK software, the Apache 2.0 license grants patent rights including RAW for those software products. However for hardware implementations CineForm RAW the licensing would be for SMPTE VC5 which grants RAW patent for custom implementations. Very roughly, the license for VC5/CineForm RAW cameras is $20 per camera in the professional market.


So not "free" to do Cineform RAW on a camera at all, and you give away sales figures that you may not want to share. Plus, guess what - red can still come after you because vc5 doesn't cover you from reds patents.

It's also not as "open" as people think and still requires some reverse engineering to actually use:

https://medium.com/@kierank_/reverse-en ... 11312bfe1c
Online
User avatar

rick.lang

  • Posts: 11092
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 5:33 am

Fascinating, Steven. Can we put this one to bed now as Cineform is never going to be relevant to BMD? Or move these discussions to the Off Topic section?


Sent from my iPhone using Tapatalk
Rick Lang
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 6:03 am

Given its history, Cineform has been relegated to a niche mezzanine codec, and Cineform raw is probably now just a historical footnote.

Braw pretty much took care of that, even if Cineform had a chance before IBC, BMD's announcement took care of that.

Sent from my H1T1000 using Tapatalk
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 7:24 am

So, $20 a camera sounds fine, and considerably better than the deal I was offered 14 or so years ago. So, if that $20 covers you for everything (presuming there is a patent pool of different patent holders) then that sounds OK.

Cineform without Bayer raw is still relevant, and you could use it in the same way Braw uses JPEG to encode Bayer into it. Instantly Braw might be better off using cineform component video compression to encode instead.
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Offline

MishaEngel

  • Posts: 943
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: BRAW? So 3:1 compression is RAW????

PostSat Mar 09, 2019 2:11 pm

Codecs determine what kind of hardware we need to post process our video files.

BRAW gave us an extra couple of years on our written off workstations (2012 i7-3960x).
Cineform gave us the same in PP and currently in Davinci Resolve.

At the moment only Kinefinity is using a dialect of CineformRAW (KineRAW) and compressed cDNG.

Only RED and Sony benefit from the current patents that forced BM to switch to BRAW (and now they bang their heads because BRAW turned out to be great in every way).

ARRI users see compressed RAW as inferiour (and they are right) or use ProRes (really smart move from Apple) when they think that it's good enough.

Now that other camera manufactures have seen that you can break the RED/Sony patent they will make their own almost RAW codecs (Canon, Pana, etc.. and maybe even ARRI) and make the RED/Sony patent irrelevant.

An other thing that is changing the industry is that Storage becomes really cheap(flash and spinning) and with AMD back in the race (together with ARM and FPGA's) compute power is massive and cheap to.
Offline

beachcomberfilms

  • Posts: 6
  • Joined: Sun Aug 11, 2019 11:18 pm
  • Real Name: Dean Mountford

Re: BRAW? So 3:1 compression is RAW????

PostThu Aug 15, 2019 7:34 pm

There are two kinds of compression... Lossy vs Lossless...

JPG is an example of lossy compression - it discards data (leading to compression artifacts)

ZIP files are an example of non-lossy compression, it compresses data to take up less file space but all the bits remain in the encoded file (e.g. TIF can be uncompressed or LZW compressed)

Someone from BMD would need to comment if their compression is lossy or lossless compression.
Offline
User avatar

Rakesh Malik

  • Posts: 2549
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Lynnwood, WA

Re: BRAW? So 3:1 compression is RAW????

PostThu Aug 15, 2019 10:41 pm

beachcomberfilms wrote:Someone from BMD would need to comment if their compression is lossy or lossless compression.


Braw is lossy. That's why it's also scalable. I believe that if you dial it down to the lowest possible compression target that it's nigh lossless, but that I'm not sure about; the one time I worked with it, we had the compression set to about the middle of the range of ratios available and got quite nice results from it.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
Alienware M15 Hexacore + Radeon VII + Razer Core
HP Spectre x360 Kaby Lake-R
Offline

Wayne Steven

  • Posts: 1954
  • Joined: Thu Aug 01, 2013 3:58 am
  • Location: Earth

Re: BRAW? So 3:1 compression is RAW????

PostFri Aug 16, 2019 9:46 am

A number of examples showed real detail scrubbed out. Such as faint lines on the bank note example. While bank notes don't matter, it shows what the codec is prepared to do.
Often people deceive themselves so much they do not understand, even when the truth is explained to them.
Next

Return to Cinematography

Who is online

Users browsing this forum: CedricLarvoire, Nathan_H and 23 guests