Page 3 of 3

Re: ProRes RAW for UMP

PostPosted: Fri Apr 13, 2018 6:35 pm
by Frank Glencairn
roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.

Mind you - This is a CPU from 2012!

Even a decent midrange CPU from last year, runs in circles around that ancient i7.

Unless you don't bring a knife to a gun fight - i.e. a under-powered laptop - working with DNGs should be zero problem.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 13, 2018 6:54 pm
by roger.magnusson
That's a 46% clock frequency increase compared to my 2.4GHz.

I'll confess that it's been a while since I had only the single six-core. I now have two but maybe BMD has optimized the DNG decompression considerably in the later v14 versions.

Edit: Interestingly, when Intel first patched the Spectre and Meltdown security vulnerabilities I installed the updated bios for my HP Z840. DNG decompression performance completely tanked, and HP pulled the update.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 13, 2018 8:14 pm
by Frank Glencairn
I never do any updates ever. But I also never have an workstation online.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 13, 2018 11:43 pm
by Robert Niessner
Frank Glencairn wrote:
roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.


Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz

My biggest grieve with cDNG was the offloading process where you could never copy any faster than half the speed a slow HD could write - and I am writing to an 8 disk RAID-5 array.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 2:47 am
by Jamie LeJeune
Frank Glencairn wrote:How is it more convenient?

For now, compatibility is the worst, only FCPx can deal with ProRes Raw (that might change though).

With DNGs the CPU is totally bored, not much to do,
and even a 3 year old midrange gamer GPU runs in circles around DNGs.

So no, and no.

I completely agree!
Presenting facts like this sadly seems to be a losing battle. Most at NAB were dazzled by the words "ProResRAW" and clamoring to have it on BMD cameras.

I fully concede that ProResRAW might someday be the better option, but that's going to require widespread support by numerous software apps on both Mac and Windows, which could take many years to develop. The original flavors of ProRes are now almost universally supported, but it took years to for that to be the case and those original ProRes codecs had the benefit of being useful as both an intermediate codec and (eventually) as a delivery codec. ProResRAW, because it is just raw sensor data, can only be an acquisition codec.
Heck, Apple hasn't even made it so that ProResRAW files can be played in the Finder. It's the same with other raw formats, but you'd think Apple would have solved that limitation on release.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 4:11 am
by Wayne Steven
I wouldn't worry too much. Because ProRes is so widely adopted itself, I imagine they will be eager to quickly adopt ProRes raw. Lots of cameras with it would help of course.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 11:31 am
by Andrew Kolakowski
Robert Niessner wrote:
Frank Glencairn wrote:
roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.


Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz

My biggest grieve with cDNG was the offloading process where you could never copy any faster than half the speed a slow HD could write - and I am writing to an 8 disk RAID-5 array.


Try using some special copy tool which allows you to do custom buffering. It may speed up things.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 1:45 pm
by Tommaso Alvisi
Frank please try to be reasonable and not being the conservative of video ;-P

1) having a gazillion individual frames is nonsense. it's not how we think, it's not how copy can be optimized both for local copies or network ones, copying / backup algos doesn't want to constantly open and close these gazillion files. On my gigabit, uploading on Google Drive is like 30 (thirty) time slower!!!!!!!!! Same line same everything just using a 20gb prores mov versus 20gb of a gazillion of small DNGs

2) compatibility is will easily and very very soon far surpass the CinemaDNG "fake" standard adoption, both in post workflows AND as in-camera acquisition format, EVERY manufacturer / brand that already uses and licenses ProRes will get on boardin the coming weeks/months...

3) decoding speed is REALLY important and it's a 3/6x advantage which is HUGE. you always need spare cpu cycles for effects (other nodes in our case) and to keep fps high (ie smooth playback), this is especially true with evolving GPUs that often in some instances the bottleneck is CPU so having more CPU power is a GODSEND

My 2c.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 3:46 pm
by Jamie LeJeune
Tommaso Alvisi wrote:3) decoding speed is REALLY important and it's a 3/6x advantage which is HUGE

Very curious where you're getting those numbers. I haven't seen any confirmed speed comparison for ProResRAW against any form of cDNG. Even against Canon Cinema Raw Lite, ProResRAW only has a 1.5x speed advantage.

https://www.apple.com/final-cut-pro/docs/Apple_ProRes_RAW_White_Paper.pdf

I'm a dedicated Mac user and FCPX is my favorite NLE. Most people would say I'm an Apple fanboy. However, I also know that their "reality distortion field" is crazy strong, so it's always best to stick to independently proven facts rather than marketing hype when evaluating their hardware and software.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 3:48 pm
by Andrew Kolakowski
Tommaso Alvisi wrote:1) having a gazillion individual frames is nonsense. it's not how we think, it's not how copy can be optimized both for local copies or network ones, copying / backup algos doesn't want to constantly open and close these gazillion files. On my gigabit, uploading on Google Drive is like 30 (thirty) time slower!!!!!!!!! Same line same everything just using a 20gb prores mov versus 20gb of a gazillion of small DNGs
...


Use rclone for google drive backups. Use eg. 64M or 128M buffer and it should be way faster (should go at about your line speed).

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 5:55 pm
by Tommaso Alvisi
Andrew Kolakowski wrote:Use rclone for google drive backups. Use eg. 64M or 128M buffer and it should be way faster (should go at about your line speed).


I use rclone, forklift, you name it I tried everything with various cache settings...minor improvement at best...

The problem is not on my side, it's their servers/infrastructure which perform poorly with small files, like many other sw and such, even Hedge for examble which uses buffering and smart cache algos is much slower with small files...it's normal, nothing new...

guys I' don't understand...are you paid to hold back everything new???

sometimes this approach seems almost troll-like...just for the sake of being in disagreement...

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 6:31 pm
by Andrew Kolakowski
Image sequence will always be more problematic than video file, but there are ways of making it performing better (buffers, transfers slots etc).
Just set --transfers 12 in rclone and this will most likely saturate your internet line. If not raise number even more to eg. 24. In this case image sequence can actually help you to saturate line speed compared to single file. I had no problem to saturate my 250Mbit line with DPX sequence going to google drive. If it doesn't happen means your storage doesn't handle image sequence well. It's also best to do transfer directly from your raid, not over network share.
I don't argue that image sequences cause more problems, but in this particular case (sending files to cloud storage) fact that you are dealing with image sequence can be actually beneficial.

Image sequence also has its advantages. Fact that in case of issue you may loose just 1 frame or just a small part of whole recording/rendering is one of them. Another thing is ability to easily render many frames at the same time.
In case of BM they can also record into 2 cards which means bandwitdth is 2x bigger so 4K RAW 60p is possible.

Re: ProRes RAW for UMP

PostPosted: Sat Apr 14, 2018 6:52 pm
by Andrew Kolakowski
Tommaso Alvisi wrote:guys I' don't understand...are you paid to hold back everything new???


What is new here? I don't see anything new here. You mean record to ProRes RAW mov?
It's up to camera manufacture to decide if they want to record into sequence or some video file.
One of the reason why BM uses image sequence can be design choice dictated by costs. Recording to single file inside camera will be most likely more difficult and probably needs more memory. Not 100% sure, so not going to argue here.
I persoanlly would also prefer single file out of camera, but this is not up to my wish. You have now sequences, so the best what you can do is to optimise the way how you deal with it and hope for change, eg, ProRes RAW mov or DNG inside MOV or MXF.

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 5:08 pm
by Janis Lionel
The reason I would switch to PR RAW is very simple:

CDNG is great. Though not great for RAID arrays. It works much faster on my SSD than on my media RAID. RAID is performing much better with large files.

So either CDNG will get a container like .mov or I would be down for ProRes RAW rather than spending a lot of money for SSDs.

Cheers

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 6:08 pm
by Tommaso Alvisi
Janis Lionel wrote:The reason I would switch to PR RAW is very simple:

CDNG is great. Though not great for RAID arrays. It works much faster on my SSD than on my media RAID. RAID is performing much better with large files.

So either CDNG will get a container like .mov or I would be down for ProRes RAW rather than spending a lot of money for SSDs.

Cheers


ditto, thanks Janis

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 6:13 pm
by Tommaso Alvisi
Andrew Kolakowski wrote:Image sequence will always be more problematic than video file, but there are ways of making it performing better (buffers, transfers slots etc).
Just set --transfers 12 in rclone and this will most likely saturate your internet line. If not raise number even more to eg. 24. In this case image sequence can actually help you to saturate line speed compared to single file. I had no problem to saturate my 250Mbit line with DPX sequence going to google drive. If it doesn't happen means your storage doesn't handle image sequence well. It's also best to do transfer directly from your raid, not over network share.
I don't argue that image sequences cause more problems, but in this particular case (sending files to cloud storage) fact that you are dealing with image sequence can be actually beneficial.

Image sequence also has its advantages. Fact that in case of issue you may loose just 1 frame or just a small part of whole recording/rendering is one of them. Another thing is ability to easily render many frames at the same time.
In case of BM they can also record into 2 cards which means bandwitdth is 2x bigger so 4K RAW 60p is possible.


I appreciate you wanting to help, but believe me, I'm using VERY fast storage as source, r0 of ssds and r0 of hdds, r5, r6 thunderbolt 3 arrays, you name it...on different oses and different hardware...I'm an user of rclone from the beginning and I know it well and believe me when I say to you that it's MUCH slower with a gazillion files. Another thing is now Google Drive is VERY picky nowadays on the number of connections so you actually have to set limits in rclone if you go up a lot with -transfers. For example in case Google Drive bans you (which as temp bans is basically daily nowadays where they enforce the 750GB daily limit...) then -checks with small files takes ages vs a big file...

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 6:47 pm
by Andrew Kolakowski
750GB/24h ban has not much to do with number of transfers. Google is full of different limits. It has to be as no one can ever give for everyone unlimited storage :)

I've actually tried it and had no issue at all to saturate one of my lines (250Mbit) with DPX sequence upload. Sometimes I upload few terabytes per night, so I also know rclone and gdrive quite well :)

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 7:54 pm
by Janis Lionel
Andrew Kolakowski wrote:
Tommaso Alvisi wrote:guys I' don't understand...are you paid to hold back everything new???


I persoanlly would also prefer single file out of camera, but this is not up to my wish. You have now sequences, so the best what you can do is to optimise the way how you deal with it and hope for change, eg, ProRes RAW mov or DNG inside MOV or MXF.


Either BMD or someone else should provide a tool to convert a cdng image seuqence into a container. Of course it would be best to have it out of camera!

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 8:21 pm
by Andrew Kolakowski
We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 9:40 pm
by Janis Lionel
Andrew Kolakowski wrote:We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.


Very true! So come on BMD!

Re: ProRes RAW for UMP

PostPosted: Sun Apr 15, 2018 10:31 pm
by Cary Knoop
Janis Lionel wrote:
Andrew Kolakowski wrote:We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.


Very true! So come on BMD!

Yes, and call it ProBest! :)

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 9:20 am
by Mihail Moskov
There is MXF mapping already defined in the CinemaDNG spec, should someone decide to implement single file containers. No need to invent stuff. :)

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 9:53 am
by Andrew Kolakowski
Yep, but this bring its own issues, so it's BM decision.
For me there are bigger problems with cDNG. Who is maintaining this format? Looks like Adobe lost its interest, BM compressed version is supported almost only by BM.
Format seem to be patch over another patch based on old technologies. There is no standardization either- just an open spec which is not properly maintained by anyone(?).

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 10:46 am
by Janis Lionel
Andrew Kolakowski wrote:Yep, but this bring its own issues, so it's BM decision.
For me there are bigger problems with cDNG. Who is maintaining this format? Looks like Adobe lost its interest, BM compressed version is supported almost only by BM and it's very inefficient (jpeg compression?).
Format seem to be patch over another patch based on old technologies. There is no standardization either- just an open spec which is not properly maintained by anyone(?).


Why is the compression bad? Isn't the data rate of 3:1 / 4:1 lower than ProRes 444?

I don't think Adobe lost interest. I think it's rather a tactical decision since Davinci Resolve is becoming a major player in editing/FX and rival vor Premiere Pro / AE. And BMD is one of the few companies pushing cDNG.

ProRes is maintained: but we see the disatvantage of that - Apple holds the torch and is very restrictive, i.e. not encoding for Windows.

I always pleged for cineform..

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 12:42 pm
by Andrew Kolakowski
Scrap all of this- it may not be exactly as efficient as ProRes, but it's around there (together with other codecs).


Ignore below- it's all wrong!
Code: Select all
For 3:1 compression (so very mild compression) quality is not very good.
At this ratio current ProRes algorithm is miles better. You can't look just at ratio- codec efficiency matters as much.
Juts note that all my tests are done against slimRAW, but it claims being at least as good as BM internal 3:1 encoding.

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 2:23 pm
by Frank Glencairn
Robert Niessner wrote:Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz



You are right 3.33GHz, it is one of the other Workstations, with a different CPU, that is overclocked.

And yes, copying is slower - but IMHO not deal breaker slow, at least for me.
There is a command line, to speed things up on NTSF drives, or you can format exFat, which is also a bit faster, since it doesn't do all that journaling. Using RoboCopy really helps a lot, when it comes to copy speed of thousands of small files. Also the copying process runs in the background, while I work on something, so I don't really care if it takes a bit longer.

So besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.

Re: ProRes RAW for UMP

PostPosted: Mon Apr 16, 2018 10:06 pm
by Jamie LeJeune
Frank Glencairn wrote: So besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.

I believe it. Apple left CDNG out of the performance comparison graphs in the marketing for ProResRAW, likely for that reason -- there is no benefit to show

Re: ProRes RAW for UMP

PostPosted: Tue Apr 17, 2018 10:33 am
by Janis Lionel
You are right 3.33GHz, it is one of the other Workstations, with a different CPU, that is overclocked.

And yes, copying is slower - but IMHO not deal breaker slow, at least for me.
There is a command line, to speed things up on NTSF drives, or you can format exFat, which is also a bit faster, since it doesn't do all that journaling. Using RoboCopy really helps a lot, when it comes to copy speed of thousands of small files. Also the copying process runs in the background, while I work on something, so I don't really care if it takes a bit longer.


Didn't know about robocopy, sounds interesting. Where can I find more info how to use it?

besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.


Are you editing on SSDs? I feel a big difference in performance editing big files on my HDD RAID 0 and the small CDNG files? Using SSDs for smaller projects - that difference is certainly gone.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 20, 2018 8:56 am
by Frank Glencairn
Janis Lionel wrote:
Didn't know about robocopy, sounds interesting. Where can I find more info how to use it?

Are you editing on SSDs? I feel a big difference in performance editing big files on my HDD RAID 0 and the small CDNG files?


Robocopy is a command line program from Microsoft, but you can google different graphical user interfaces for it, if you are not comfortable wit command lines.

I'm not editing from SSDs, since the typical size of my projects doesn't make that an option.
I'm editing in realtime from my RAID 0 anyway, and even export is often at 35 fps or more (24P project), so I don't see any benefit in SSDs here.

Re: ProRes RAW for UMP

PostPosted: Wed Apr 25, 2018 10:10 pm
by willcampbelltv
This YouTube video from NAB 2018 has a BlackMagic rep talking about how the company is in process of considering if they will be integrating ProRes RAW into Black Magic Products. Skip to 7:57


Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 4:31 am
by Uli Plank
And BM should look and consider it carefully before offering it.
There is one big difference between RedRAW compression and ProRes (if Apple's RAW is still using the same scheme): one is Wavelet based and the other DCT. Softness can look quite natural.

Higher compressing ratios in Wavelet tend to make the image softer, and I've seen DoPs using higher compression on Red in a pinch when they didn't carry enough diffusion with them.
DCT tends to introduce very defined artifacts, like mosquito noise and macro-blocking, which can produce visible harshness, which people hate in digital imagery.

I don't think you can really see the difference at 3:1, but I'd always take a 8:1 compression by Wavelet over a 8:1 ratio in DCT.
I don't doubt the advantage of compressing RAW before debayering, but can we have Cineform, please?

Just my two cents (Euro)

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 8:57 am
by Andrew Kolakowski
Yes, wavelet based codecs have bit different artefacts- as you said, mainly soften image. DCT based are more "harsh", but at those ratio values this is not going to be really visible. DNG is also DCT based and it's rather less efficient than ProRes, so quality wise ProRes RAW not going to be worse.

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 1:31 pm
by Mike Faber
This is maybe just me but I'm personally trying to stop the apple prores machine, not feed it. I'm a pc based editor and do mostly broadcast/commercial work. The work arounds I have to do to submit prores to stations is just annoying. Every second counts in my world and I'm not ever moving backwards to final cut and in all likely hood will never invest in a mac pro with their bloated prices and non-upgradable system. ProRes files edit easily enough but isn't the eye opening, dance on the table experience people make it out to be. I would prefer ProRes be able to be at least ingested into Resolve for PC but Apple likes to force your hand for their products and I wouldn't be surprised if this is their strategy to get masses back onto final cut.

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 1:57 pm
by Wayne Steven
I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 1:58 pm
by Wayne Steven
Mike. I agree, we need to think about these things. What do you propose using as a high quality capture format, instead?

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 2:50 pm
by Andrew Kolakowski
Wayne Steven wrote:I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?


Why? What stops you to compress RAW data with wavelet based codec :) This is what RED does and Cineform can do. There is nothing that special about compression RAW- it's just some data like final debayered one. If anything it may be easier to compress as it's monochromatic and we know its nature.

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 2:52 pm
by Andrew Kolakowski
Mike Faber wrote:This is maybe just me but I'm personally trying to stop the apple prores machine, not feed it. I'm a pc based editor and do mostly broadcast/commercial work. The work arounds I have to do to submit prores to stations is just annoying. Every second counts in my world and I'm not ever moving backwards to final cut and in all likely hood will never invest in a mac pro with their bloated prices and non-upgradable system. ProRes files edit easily enough but isn't the eye opening, dance on the table experience people make it out to be. I would prefer ProRes be able to be at least ingested into Resolve for PC but Apple likes to force your hand for their products and I wouldn't be surprised if this is their strategy to get masses back onto final cut.


There is one big difference here. You don't have to worry about making ProRes RAW (this happens in camera or recorder)- all what you need is decoding :) If Apple is not going to be "funny" about it then it's not a big deal that it's Apple's codec. If they will be "funny" about it then we may already stop talking about ProRes RAW.

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 6:24 pm
by Cary Knoop
Andrew Kolakowski wrote:There is nothing that special about compression RAW

Well, hold on, I know your expertise about it is far higher than mine so I am a bit confused by this.

Fourier based transforms, which are used by both DCT and Wavelet based compression methods, are effective because you can base compression levels on the frequency of regional data, i.e. high frequency regions are compressed more than low frequency regions.

However in case of RAW data the frequency is only indirectly known, unless you de-mosaic you can't readily know the frequency of the data or at least it is less accurate.

Am I wrong with that assumption?

Re: ProRes RAW for UMP

PostPosted: Thu Apr 26, 2018 7:57 pm
by Andrew Kolakowski
Have you seen RAW image?

Image

It's nothing different than "normal" image. It just "a picture" with many squares, but nothing really stops you to apply same "compression" logic as with debayered one. We know its nature, so maybe there is a way to make optimised codec, but all current codecs compress RAW images very well. What would be interesting is a study how different compression on RAW affects final debayered quality. I have not seen any, just done my own quick tests.

You can take any BM DNG sequence, convert with dcraw to "normal" 16bit TIFF (which will hold RAW data and look like sample above), compress this with any codec you want and compare with original and eg. slimRAW DNG compression. You can measure eg. PSNR on RAW data same way as on normal image and it will still have meaning and give you useful information. ProRes XQ (at about 4.5:1) compresses RAW images very well.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 27, 2018 4:39 am
by Wayne Steven
Andrew Kolakowski wrote:
Wayne Steven wrote:I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?


Why? What stops you to compress RAW data with wavelet based codec :) This is what RED does and Cineform can do. There is nothing that special about compression RAW- it's just some data like final debayered one. If anything it may be easier to compress as it's monochromatic and we know its nature.


Andrew, it helps to understand the structure of the concepts involved. Bayer is a lossy way to do imaging, which means it starts out less data and debayering can produce a more information sparse result that is easier to compress. Another concept on Bayer is that low pass filtering can be pretty drastic spreading light between neighbouring pixels (even a long way away). This softens the image, and makes it more compressible in Bayer or regular pixel encoding. So, comparing apples to apples and oranges to oranges, helps determine how compressible it is. So no low pass filter Bayer compressed against no low pass filtering Bayer debayered then compressed, gives an idea of compressibility. Bayer is still likely to be smaller, as it's smaller to start with and the debayered tries to remake missing data that has to be compressed ontop. It is of course likely going look more compressible to three chip stuff, as that has more information to start with.

Now, another concept. When Dave at cineform did cineform raw he simply compressed the color channels separately as waves, and I forget what conversion to luminance chroma spaces. It was simple. But it means it's more waves that pass through pixel points rather then pixels.

But the obvious thing with Raw Bayer is that you preseve original information to reconstitute the missing colour information as best as you can. Debayering in camera before compression can produce worse results. However, compressed debayered is still usable.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 27, 2018 9:44 am
by Andrew Kolakowski
I understand what you are saying, but don't get your final point :)

As I said- only missing bit is study how different compression methods (and ratios) affect final debayered image. From what I see it's not much different than for debayered one. 3:1 DNG vs. lossless one on RAW image has 54dB PSNR. Debayer same images and PSNR on final image is 53dB (exported as 12bit DPX). Funny enough 7:1 DNG is 42dB on RAW vs. 44dB on final debayered image (bit strange). Even at 7:1 DNG RAW you can't tell original vs. compressed on final image at 100% frame size when switching between both and looking at them from 15cm. I don't see it being big deal shooting at 4:1 or BM adding even higher compression modes (and enabling higher fps if possible). Shooting at 7:1 may not be desired when your content will be shown at cinema (or some VFX work), but for anything else it will be basically as good as uncompressed (it will be most likely compressed to the point where any possibly still visible difference even on eg. 75inch TV will be gone).
If you don't do low pass filtering not only RAW will be harder to compress but final image also, so this is really irrelevant.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 27, 2018 4:32 pm
by Rakesh Malik
Andrew Kolakowski wrote:As I said- only missing bit is study how different compression methods (and ratios) affect final debayered image. From what I see it's not much different than for debayered one. 3:1 DNG vs. lossless one on RAW image has 54dB PSNR.


As far as the final image goes, there's really not going to be any difference as long as the codec is good, which is almost redundant with "modern" these days. Space wise there is some additional savings from being a single file instead of a collection, both from a file system allocation point of view (fewer wasted blocks due to an image exceeding a block but just barely), and only one copy of the Bayer pattern bit map for an entire clip rather than one for every frame as is the case for cDNG.

Re: ProRes RAW for UMP

PostPosted: Fri Apr 27, 2018 6:18 pm
by Andrew Kolakowski
Last message had no much reference to ProRes RAW itself. It's about RAW vs. debayered image- if they are any different in nature/need different coded design etc.
Saved bits on image sequence vs. MOV is really meaningless.
RAW metadata will be probably repeated on ProRes RAW frame headers levels (not only MOV headers) in case of trimming needs etc. It's tiny compared to actual video data.