ProRes RAW for UMP

The place for questions about shooting with Blackmagic Cameras.
  • Author
  • Message
Offline
User avatar

Frank Glencairn

  • Posts: 1610
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: ProRes RAW for UMP

PostFri Apr 13, 2018 6:35 pm

roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.

Mind you - This is a CPU from 2012!

Even a decent midrange CPU from last year, runs in circles around that ancient i7.

Unless you don't bring a knife to a gun fight - i.e. a under-powered laptop - working with DNGs should be zero problem.
http://frankglencairn.wordpress.com/

I told you so :-)
Offline

roger.magnusson

  • Posts: 400
  • Joined: Wed Sep 23, 2015 4:58 pm
  • Location: Sweden

Re: ProRes RAW for UMP

PostFri Apr 13, 2018 6:54 pm

That's a 46% clock frequency increase compared to my 2.4GHz.

I'll confess that it's been a while since I had only the single six-core. I now have two but maybe BMD has optimized the DNG decompression considerably in the later v14 versions.

Edit: Interestingly, when Intel first patched the Spectre and Meltdown security vulnerabilities I installed the updated bios for my HP Z840. DNG decompression performance completely tanked, and HP pulled the update.
Offline
User avatar

Frank Glencairn

  • Posts: 1610
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: ProRes RAW for UMP

PostFri Apr 13, 2018 8:14 pm

I never do any updates ever. But I also never have an workstation online.
http://frankglencairn.wordpress.com/

I told you so :-)
Offline
User avatar

Robert Niessner

  • Posts: 1482
  • Joined: Thu Feb 21, 2013 9:51 am
  • Location: Graz, Austria

Re: ProRes RAW for UMP

PostFri Apr 13, 2018 11:43 pm

Frank Glencairn wrote:
roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.


Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz

My biggest grieve with cDNG was the offloading process where you could never copy any faster than half the speed a slow HD could write - and I am writing to an 8 disk RAID-5 array.
Robert Niessner
LAUFBILDkommission
Graz / Austria
---------------------------------------------------------------------
Blackmagic Camera Blog (German):
http://laufbildkommission.wordpress.com

Read the blog in English via Google Translate:
http://tinyurl.com/pjf6a3m
Offline
User avatar

Jamie LeJeune

  • Posts: 859
  • Joined: Tue Feb 26, 2013 4:33 am
  • Location: San Francisco

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 2:47 am

Frank Glencairn wrote:How is it more convenient?

For now, compatibility is the worst, only FCPx can deal with ProRes Raw (that might change though).

With DNGs the CPU is totally bored, not much to do,
and even a 3 year old midrange gamer GPU runs in circles around DNGs.

So no, and no.

I completely agree!
Presenting facts like this sadly seems to be a losing battle. Most at NAB were dazzled by the words "ProResRAW" and clamoring to have it on BMD cameras.

I fully concede that ProResRAW might someday be the better option, but that's going to require widespread support by numerous software apps on both Mac and Windows, which could take many years to develop. The original flavors of ProRes are now almost universally supported, but it took years to for that to be the case and those original ProRes codecs had the benefit of being useful as both an intermediate codec and (eventually) as a delivery codec. ProResRAW, because it is just raw sensor data, can only be an acquisition codec.
Heck, Apple hasn't even made it so that ProResRAW files can be played in the Finder. It's the same with other raw formats, but you'd think Apple would have solved that limitation on release.
www.cinedocs.com
http://www.imdb.com/name/nm4601572/
Offline

Wayne Steven

  • Posts: 755
  • Joined: Thu Aug 01, 2013 3:58 am

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 4:11 am

I wouldn't worry too much. Because ProRes is so widely adopted itself, I imagine they will be eager to quickly adopt ProRes raw. Lots of cameras with it would help of course.
Last edited by Wayne Steven on Sun Apr 15, 2018 1:50 pm, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 11:31 am

Robert Niessner wrote:
Frank Glencairn wrote:
roger.magnusson wrote:Depends on the CPU. :)

A six core CPU or less at about 2.4GHz will have trouble with 4K lossless compressed raw. Uncompressed raw should be quite easy but I think all BMD cameras have lossless compression now.


Oh please - I have a i7 6 core Gulftown 980, overclocked to 3.4 MhZ here, that runs at maybe 50% when playing 4k lossless compressed raw.


Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz

My biggest grieve with cDNG was the offloading process where you could never copy any faster than half the speed a slow HD could write - and I am writing to an 8 disk RAID-5 array.


Try using some special copy tool which allows you to do custom buffering. It may speed up things.
Offline
User avatar

Tommaso Alvisi

  • Posts: 108
  • Joined: Wed Aug 22, 2012 7:53 am
  • Location: ITALY

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 1:45 pm

Frank please try to be reasonable and not being the conservative of video ;-P

1) having a gazillion individual frames is nonsense. it's not how we think, it's not how copy can be optimized both for local copies or network ones, copying / backup algos doesn't want to constantly open and close these gazillion files. On my gigabit, uploading on Google Drive is like 30 (thirty) time slower!!!!!!!!! Same line same everything just using a 20gb prores mov versus 20gb of a gazillion of small DNGs

2) compatibility is will easily and very very soon far surpass the CinemaDNG "fake" standard adoption, both in post workflows AND as in-camera acquisition format, EVERY manufacturer / brand that already uses and licenses ProRes will get on boardin the coming weeks/months...

3) decoding speed is REALLY important and it's a 3/6x advantage which is HUGE. you always need spare cpu cycles for effects (other nodes in our case) and to keep fps high (ie smooth playback), this is especially true with evolving GPUs that often in some instances the bottleneck is CPU so having more CPU power is a GODSEND

My 2c.
Offline
User avatar

Jamie LeJeune

  • Posts: 859
  • Joined: Tue Feb 26, 2013 4:33 am
  • Location: San Francisco

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 3:46 pm

Tommaso Alvisi wrote:3) decoding speed is REALLY important and it's a 3/6x advantage which is HUGE

Very curious where you're getting those numbers. I haven't seen any confirmed speed comparison for ProResRAW against any form of cDNG. Even against Canon Cinema Raw Lite, ProResRAW only has a 1.5x speed advantage.

https://www.apple.com/final-cut-pro/docs/Apple_ProRes_RAW_White_Paper.pdf

I'm a dedicated Mac user and FCPX is my favorite NLE. Most people would say I'm an Apple fanboy. However, I also know that their "reality distortion field" is crazy strong, so it's always best to stick to independently proven facts rather than marketing hype when evaluating their hardware and software.
www.cinedocs.com
http://www.imdb.com/name/nm4601572/
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 3:48 pm

Tommaso Alvisi wrote:1) having a gazillion individual frames is nonsense. it's not how we think, it's not how copy can be optimized both for local copies or network ones, copying / backup algos doesn't want to constantly open and close these gazillion files. On my gigabit, uploading on Google Drive is like 30 (thirty) time slower!!!!!!!!! Same line same everything just using a 20gb prores mov versus 20gb of a gazillion of small DNGs
...


Use rclone for google drive backups. Use eg. 64M or 128M buffer and it should be way faster (should go at about your line speed).
Offline
User avatar

Tommaso Alvisi

  • Posts: 108
  • Joined: Wed Aug 22, 2012 7:53 am
  • Location: ITALY

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 5:55 pm

Andrew Kolakowski wrote:Use rclone for google drive backups. Use eg. 64M or 128M buffer and it should be way faster (should go at about your line speed).


I use rclone, forklift, you name it I tried everything with various cache settings...minor improvement at best...

The problem is not on my side, it's their servers/infrastructure which perform poorly with small files, like many other sw and such, even Hedge for examble which uses buffering and smart cache algos is much slower with small files...it's normal, nothing new...

guys I' don't understand...are you paid to hold back everything new???

sometimes this approach seems almost troll-like...just for the sake of being in disagreement...
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 6:31 pm

Image sequence will always be more problematic than video file, but there are ways of making it performing better (buffers, transfers slots etc).
Just set --transfers 12 in rclone and this will most likely saturate your internet line. If not raise number even more to eg. 24. In this case image sequence can actually help you to saturate line speed compared to single file. I had no problem to saturate my 250Mbit line with DPX sequence going to google drive. If it doesn't happen means your storage doesn't handle image sequence well. It's also best to do transfer directly from your raid, not over network share.
I don't argue that image sequences cause more problems, but in this particular case (sending files to cloud storage) fact that you are dealing with image sequence can be actually beneficial.

Image sequence also has its advantages. Fact that in case of issue you may loose just 1 frame or just a small part of whole recording/rendering is one of them. Another thing is ability to easily render many frames at the same time.
In case of BM they can also record into 2 cards which means bandwitdth is 2x bigger so 4K RAW 60p is possible.
Last edited by Andrew Kolakowski on Sun Apr 15, 2018 11:11 am, edited 3 times in total.
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSat Apr 14, 2018 6:52 pm

Tommaso Alvisi wrote:guys I' don't understand...are you paid to hold back everything new???


What is new here? I don't see anything new here. You mean record to ProRes RAW mov?
It's up to camera manufacture to decide if they want to record into sequence or some video file.
One of the reason why BM uses image sequence can be design choice dictated by costs. Recording to single file inside camera will be most likely more difficult and probably needs more memory. Not 100% sure, so not going to argue here.
I persoanlly would also prefer single file out of camera, but this is not up to my wish. You have now sequences, so the best what you can do is to optimise the way how you deal with it and hope for change, eg, ProRes RAW mov or DNG inside MOV or MXF.
Offline

Janis Lionel

  • Posts: 117
  • Joined: Mon Apr 04, 2016 8:09 pm

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 5:08 pm

The reason I would switch to PR RAW is very simple:

CDNG is great. Though not great for RAID arrays. It works much faster on my SSD than on my media RAID. RAID is performing much better with large files.

So either CDNG will get a container like .mov or I would be down for ProRes RAW rather than spending a lot of money for SSDs.

Cheers
--
Windows 10 / i4930k @4.3Ghz / 32GB RAM / GTX 1080 / 12TB RAID 0 (4 x 3 TB) / Mini Monitor 4K / Eizo CG247x / Mainly working with CDNG 4:1
Offline
User avatar

Tommaso Alvisi

  • Posts: 108
  • Joined: Wed Aug 22, 2012 7:53 am
  • Location: ITALY

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 6:08 pm

Janis Lionel wrote:The reason I would switch to PR RAW is very simple:

CDNG is great. Though not great for RAID arrays. It works much faster on my SSD than on my media RAID. RAID is performing much better with large files.

So either CDNG will get a container like .mov or I would be down for ProRes RAW rather than spending a lot of money for SSDs.

Cheers


ditto, thanks Janis
Offline
User avatar

Tommaso Alvisi

  • Posts: 108
  • Joined: Wed Aug 22, 2012 7:53 am
  • Location: ITALY

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 6:13 pm

Andrew Kolakowski wrote:Image sequence will always be more problematic than video file, but there are ways of making it performing better (buffers, transfers slots etc).
Just set --transfers 12 in rclone and this will most likely saturate your internet line. If not raise number even more to eg. 24. In this case image sequence can actually help you to saturate line speed compared to single file. I had no problem to saturate my 250Mbit line with DPX sequence going to google drive. If it doesn't happen means your storage doesn't handle image sequence well. It's also best to do transfer directly from your raid, not over network share.
I don't argue that image sequences cause more problems, but in this particular case (sending files to cloud storage) fact that you are dealing with image sequence can be actually beneficial.

Image sequence also has its advantages. Fact that in case of issue you may loose just 1 frame or just a small part of whole recording/rendering is one of them. Another thing is ability to easily render many frames at the same time.
In case of BM they can also record into 2 cards which means bandwitdth is 2x bigger so 4K RAW 60p is possible.


I appreciate you wanting to help, but believe me, I'm using VERY fast storage as source, r0 of ssds and r0 of hdds, r5, r6 thunderbolt 3 arrays, you name it...on different oses and different hardware...I'm an user of rclone from the beginning and I know it well and believe me when I say to you that it's MUCH slower with a gazillion files. Another thing is now Google Drive is VERY picky nowadays on the number of connections so you actually have to set limits in rclone if you go up a lot with -transfers. For example in case Google Drive bans you (which as temp bans is basically daily nowadays where they enforce the 750GB daily limit...) then -checks with small files takes ages vs a big file...
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 6:47 pm

750GB/24h ban has not much to do with number of transfers. Google is full of different limits. It has to be as no one can ever give for everyone unlimited storage :)

I've actually tried it and had no issue at all to saturate one of my lines (250Mbit) with DPX sequence upload. Sometimes I upload few terabytes per night, so I also know rclone and gdrive quite well :)
Offline

Janis Lionel

  • Posts: 117
  • Joined: Mon Apr 04, 2016 8:09 pm

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 7:54 pm

Andrew Kolakowski wrote:
Tommaso Alvisi wrote:guys I' don't understand...are you paid to hold back everything new???


I persoanlly would also prefer single file out of camera, but this is not up to my wish. You have now sequences, so the best what you can do is to optimise the way how you deal with it and hope for change, eg, ProRes RAW mov or DNG inside MOV or MXF.


Either BMD or someone else should provide a tool to convert a cdng image seuqence into a container. Of course it would be best to have it out of camera!
--
Windows 10 / i4930k @4.3Ghz / 32GB RAM / GTX 1080 / 12TB RAID 0 (4 x 3 TB) / Mini Monitor 4K / Eizo CG247x / Mainly working with CDNG 4:1
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 8:21 pm

We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.
Offline

Janis Lionel

  • Posts: 117
  • Joined: Mon Apr 04, 2016 8:09 pm

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 9:40 pm

Andrew Kolakowski wrote:We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.


Very true! So come on BMD!
--
Windows 10 / i4930k @4.3Ghz / 32GB RAM / GTX 1080 / 12TB RAID 0 (4 x 3 TB) / Mini Monitor 4K / Eizo CG247x / Mainly working with CDNG 4:1
Offline
User avatar

Cary Knoop

  • Posts: 509
  • Joined: Sun Mar 12, 2017 6:35 pm
  • Location: Newark, CA USA

Re: ProRes RAW for UMP

PostSun Apr 15, 2018 10:31 pm

Janis Lionel wrote:
Andrew Kolakowski wrote:We can pack DNG into something, but without support for it it will be useless. It has to be BM as this would also help to "enforce" support in other apps than Resolve.

You can always zip/rar it (even with 0 compression), so you will have 1 big file.


Very true! So come on BMD!

Yes, and call it ProBest! :)
Offline

Mihail Moskov

  • Posts: 37
  • Joined: Wed Jun 24, 2015 11:18 am

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 9:20 am

There is MXF mapping already defined in the CinemaDNG spec, should someone decide to implement single file containers. No need to invent stuff. :)
www.shutterangle.com
The science & magic of shooting moving pictures

www.slimraw.com
A fast CinemaDNG compressor
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 9:53 am

Yep, but this bring its own issues, so it's BM decision.
For me there are bigger problems with cDNG. Who is maintaining this format? Looks like Adobe lost its interest, BM compressed version is supported almost only by BM.
Format seem to be patch over another patch based on old technologies. There is no standardization either- just an open spec which is not properly maintained by anyone(?).
Last edited by Andrew Kolakowski on Mon Apr 16, 2018 11:34 pm, edited 1 time in total.
Offline

Janis Lionel

  • Posts: 117
  • Joined: Mon Apr 04, 2016 8:09 pm

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 10:46 am

Andrew Kolakowski wrote:Yep, but this bring its own issues, so it's BM decision.
For me there are bigger problems with cDNG. Who is maintaining this format? Looks like Adobe lost its interest, BM compressed version is supported almost only by BM and it's very inefficient (jpeg compression?).
Format seem to be patch over another patch based on old technologies. There is no standardization either- just an open spec which is not properly maintained by anyone(?).


Why is the compression bad? Isn't the data rate of 3:1 / 4:1 lower than ProRes 444?

I don't think Adobe lost interest. I think it's rather a tactical decision since Davinci Resolve is becoming a major player in editing/FX and rival vor Premiere Pro / AE. And BMD is one of the few companies pushing cDNG.

ProRes is maintained: but we see the disatvantage of that - Apple holds the torch and is very restrictive, i.e. not encoding for Windows.

I always pleged for cineform..
--
Windows 10 / i4930k @4.3Ghz / 32GB RAM / GTX 1080 / 12TB RAID 0 (4 x 3 TB) / Mini Monitor 4K / Eizo CG247x / Mainly working with CDNG 4:1
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 12:42 pm

Scrap all of this- it may not be exactly as efficient as ProRes, but it's around there (together with other codecs).


Ignore below- it's all wrong!
Code: Select all
For 3:1 compression (so very mild compression) quality is not very good.
At this ratio current ProRes algorithm is miles better. You can't look just at ratio- codec efficiency matters as much.
Juts note that all my tests are done against slimRAW, but it claims being at least as good as BM internal 3:1 encoding.
Last edited by Andrew Kolakowski on Tue Apr 17, 2018 10:21 am, edited 5 times in total.
Offline
User avatar

Frank Glencairn

  • Posts: 1610
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 2:23 pm

Robert Niessner wrote:Are you sure it is overclocked Frank? If have the same CPU in my Workstation and it has a base clock of 3.33 GHz



You are right 3.33GHz, it is one of the other Workstations, with a different CPU, that is overclocked.

And yes, copying is slower - but IMHO not deal breaker slow, at least for me.
There is a command line, to speed things up on NTSF drives, or you can format exFat, which is also a bit faster, since it doesn't do all that journaling. Using RoboCopy really helps a lot, when it comes to copy speed of thousands of small files. Also the copying process runs in the background, while I work on something, so I don't really care if it takes a bit longer.

So besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.
http://frankglencairn.wordpress.com/

I told you so :-)
Offline
User avatar

Jamie LeJeune

  • Posts: 859
  • Joined: Tue Feb 26, 2013 4:33 am
  • Location: San Francisco

Re: ProRes RAW for UMP

PostMon Apr 16, 2018 10:06 pm

Frank Glencairn wrote: So besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.

I believe it. Apple left CDNG out of the performance comparison graphs in the marketing for ProResRAW, likely for that reason -- there is no benefit to show
www.cinedocs.com
http://www.imdb.com/name/nm4601572/
Offline

Janis Lionel

  • Posts: 117
  • Joined: Mon Apr 04, 2016 8:09 pm

Re: ProRes RAW for UMP

PostTue Apr 17, 2018 10:33 am

You are right 3.33GHz, it is one of the other Workstations, with a different CPU, that is overclocked.

And yes, copying is slower - but IMHO not deal breaker slow, at least for me.
There is a command line, to speed things up on NTSF drives, or you can format exFat, which is also a bit faster, since it doesn't do all that journaling. Using RoboCopy really helps a lot, when it comes to copy speed of thousands of small files. Also the copying process runs in the background, while I work on something, so I don't really care if it takes a bit longer.


Didn't know about robocopy, sounds interesting. Where can I find more info how to use it?

besides some file copying, it still stands, once it is on your timeline, performance wise, PRR has no advantage over CDNG.


Are you editing on SSDs? I feel a big difference in performance editing big files on my HDD RAID 0 and the small CDNG files? Using SSDs for smaller projects - that difference is certainly gone.
--
Windows 10 / i4930k @4.3Ghz / 32GB RAM / GTX 1080 / 12TB RAID 0 (4 x 3 TB) / Mini Monitor 4K / Eizo CG247x / Mainly working with CDNG 4:1
Offline
User avatar

Frank Glencairn

  • Posts: 1610
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: ProRes RAW for UMP

PostFri Apr 20, 2018 8:56 am

Janis Lionel wrote:
Didn't know about robocopy, sounds interesting. Where can I find more info how to use it?

Are you editing on SSDs? I feel a big difference in performance editing big files on my HDD RAID 0 and the small CDNG files?


Robocopy is a command line program from Microsoft, but you can google different graphical user interfaces for it, if you are not comfortable wit command lines.

I'm not editing from SSDs, since the typical size of my projects doesn't make that an option.
I'm editing in realtime from my RAID 0 anyway, and even export is often at 35 fps or more (24P project), so I don't see any benefit in SSDs here.
http://frankglencairn.wordpress.com/

I told you so :-)
Offline

willcampbelltv

  • Posts: 1
  • Joined: Wed Apr 25, 2018 10:07 pm
  • Real Name: Will Campbell

Re: ProRes RAW for UMP

PostWed Apr 25, 2018 10:10 pm

This YouTube video from NAB 2018 has a BlackMagic rep talking about how the company is in process of considering if they will be integrating ProRes RAW into Black Magic Products. Skip to 7:57

Offline

Uli Plank

  • Posts: 2858
  • Joined: Fri Feb 08, 2013 2:48 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 4:31 am

And BM should look and consider it carefully before offering it.
There is one big difference between RedRAW compression and ProRes (if Apple's RAW is still using the same scheme): one is Wavelet based and the other DCT. Softness can look quite natural.

Higher compressing ratios in Wavelet tend to make the image softer, and I've seen DoPs using higher compression on Red in a pinch when they didn't carry enough diffusion with them.
DCT tends to introduce very defined artifacts, like mosquito noise and macro-blocking, which can produce visible harshness, which people hate in digital imagery.

I don't think you can really see the difference at 3:1, but I'd always take a 8:1 compression by Wavelet over a 8:1 ratio in DCT.
I don't doubt the advantage of compressing RAW before debayering, but can we have Cineform, please?

Just my two cents (Euro)
Resolve Studio and Fusion Studio
iMac 2017 Radeon Pro 580 8 GB VRAM and 32 GB RAM
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 8:57 am

Yes, wavelet based codecs have bit different artefacts- as you said, mainly soften image. DCT based are more "harsh", but at those ratio values this is not going to be really visible. DNG is also DCT based and it's rather less efficient than ProRes, so quality wise ProRes RAW not going to be worse.
Offline
User avatar

Mike Faber

  • Posts: 19
  • Joined: Wed Jan 08, 2014 1:13 pm
  • Location: Washington D.C.

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 1:31 pm

This is maybe just me but I'm personally trying to stop the apple prores machine, not feed it. I'm a pc based editor and do mostly broadcast/commercial work. The work arounds I have to do to submit prores to stations is just annoying. Every second counts in my world and I'm not ever moving backwards to final cut and in all likely hood will never invest in a mac pro with their bloated prices and non-upgradable system. ProRes files edit easily enough but isn't the eye opening, dance on the table experience people make it out to be. I would prefer ProRes be able to be at least ingested into Resolve for PC but Apple likes to force your hand for their products and I wouldn't be surprised if this is their strategy to get masses back onto final cut.
Offline

Wayne Steven

  • Posts: 755
  • Joined: Thu Aug 01, 2013 3:58 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 1:57 pm

I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?
Offline

Wayne Steven

  • Posts: 755
  • Joined: Thu Aug 01, 2013 3:58 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 1:58 pm

Mike. I agree, we need to think about these things. What do you propose using as a high quality capture format, instead?
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 2:50 pm

Wayne Steven wrote:I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?


Why? What stops you to compress RAW data with wavelet based codec :) This is what RED does and Cineform can do. There is nothing that special about compression RAW- it's just some data like final debayered one. If anything it may be easier to compress as it's monochromatic and we know its nature.
Last edited by Andrew Kolakowski on Thu Apr 26, 2018 2:54 pm, edited 1 time in total.
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 2:52 pm

Mike Faber wrote:This is maybe just me but I'm personally trying to stop the apple prores machine, not feed it. I'm a pc based editor and do mostly broadcast/commercial work. The work arounds I have to do to submit prores to stations is just annoying. Every second counts in my world and I'm not ever moving backwards to final cut and in all likely hood will never invest in a mac pro with their bloated prices and non-upgradable system. ProRes files edit easily enough but isn't the eye opening, dance on the table experience people make it out to be. I would prefer ProRes be able to be at least ingested into Resolve for PC but Apple likes to force your hand for their products and I wouldn't be surprised if this is their strategy to get masses back onto final cut.


There is one big difference here. You don't have to worry about making ProRes RAW (this happens in camera or recorder)- all what you need is decoding :) If Apple is not going to be "funny" about it then it's not a big deal that it's Apple's codec. If they will be "funny" about it then we may already stop talking about ProRes RAW.
Offline
User avatar

Cary Knoop

  • Posts: 509
  • Joined: Sun Mar 12, 2017 6:35 pm
  • Location: Newark, CA USA

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 6:24 pm

Andrew Kolakowski wrote:There is nothing that special about compression RAW

Well, hold on, I know your expertise about it is far higher than mine so I am a bit confused by this.

Fourier based transforms, which are used by both DCT and Wavelet based compression methods, are effective because you can base compression levels on the frequency of regional data, i.e. high frequency regions are compressed more than low frequency regions.

However in case of RAW data the frequency is only indirectly known, unless you de-mosaic you can't readily know the frequency of the data or at least it is less accurate.

Am I wrong with that assumption?
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostThu Apr 26, 2018 7:57 pm

Have you seen RAW image?

Image

It's nothing different than "normal" image. It just "a picture" with many squares, but nothing really stops you to apply same "compression" logic as with debayered one. We know its nature, so maybe there is a way to make optimised codec, but all current codecs compress RAW images very well. What would be interesting is a study how different compression on RAW affects final debayered quality. I have not seen any, just done my own quick tests.

You can take any BM DNG sequence, convert with dcraw to "normal" 16bit TIFF (which will hold RAW data and look like sample above), compress this with any codec you want and compare with original and eg. slimRAW DNG compression. You can measure eg. PSNR on RAW data same way as on normal image and it will still have meaning and give you useful information. ProRes XQ (at about 4.5:1) compresses RAW images very well.
Offline

Wayne Steven

  • Posts: 755
  • Joined: Thu Aug 01, 2013 3:58 am

Re: ProRes RAW for UMP

PostFri Apr 27, 2018 4:39 am

Andrew Kolakowski wrote:
Wayne Steven wrote:I remember years ago, wavelet was supposed to be double jpeg in compressibility. But the overlapping dct stuff was supposed to match it. 8:1 is pushing it.

But are you comparing the wavelet Bayer compressed against 4:4:4 or 4:2:2 debayered wavelet compressed (which should be bigger just from dealing from larger reconstituted data)? Or are you comparing 3 chip 4:2:2 or 4:4:4, which holds more information than raw and should be less soft?


Why? What stops you to compress RAW data with wavelet based codec :) This is what RED does and Cineform can do. There is nothing that special about compression RAW- it's just some data like final debayered one. If anything it may be easier to compress as it's monochromatic and we know its nature.


Andrew, it helps to understand the structure of the concepts involved. Bayer is a lossy way to do imaging, which means it starts out less data and debayering can produce a more information sparse result that is easier to compress. Another concept on Bayer is that low pass filtering can be pretty drastic spreading light between neighbouring pixels (even a long way away). This softens the image, and makes it more compressible in Bayer or regular pixel encoding. So, comparing apples to apples and oranges to oranges, helps determine how compressible it is. So no low pass filter Bayer compressed against no low pass filtering Bayer debayered then compressed, gives an idea of compressibility. Bayer is still likely to be smaller, as it's smaller to start with and the debayered tries to remake missing data that has to be compressed ontop. It is of course likely going look more compressible to three chip stuff, as that has more information to start with.

Now, another concept. When Dave at cineform did cineform raw he simply compressed the color channels separately as waves, and I forget what conversion to luminance chroma spaces. It was simple. But it means it's more waves that pass through pixel points rather then pixels.

But the obvious thing with Raw Bayer is that you preseve original information to reconstitute the missing colour information as best as you can. Debayering in camera before compression can produce worse results. However, compressed debayered is still usable.
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostFri Apr 27, 2018 9:44 am

I understand what you are saying, but don't get your final point :)

As I said- only missing bit is study how different compression methods (and ratios) affect final debayered image. From what I see it's not much different than for debayered one. 3:1 DNG vs. lossless one on RAW image has 54dB PSNR. Debayer same images and PSNR on final image is 53dB (exported as 12bit DPX). Funny enough 7:1 DNG is 42dB on RAW vs. 44dB on final debayered image (bit strange). Even at 7:1 DNG RAW you can't tell original vs. compressed on final image at 100% frame size when switching between both and looking at them from 15cm. I don't see it being big deal shooting at 4:1 or BM adding even higher compression modes (and enabling higher fps if possible). Shooting at 7:1 may not be desired when your content will be shown at cinema (or some VFX work), but for anything else it will be basically as good as uncompressed (it will be most likely compressed to the point where any possibly still visible difference even on eg. 75inch TV will be gone).
If you don't do low pass filtering not only RAW will be harder to compress but final image also, so this is really irrelevant.
Offline
User avatar

Rakesh Malik

  • Posts: 1960
  • Joined: Fri Dec 07, 2012 1:01 am
  • Location: Tacoma, WA

Re: ProRes RAW for UMP

PostFri Apr 27, 2018 4:32 pm

Andrew Kolakowski wrote:As I said- only missing bit is study how different compression methods (and ratios) affect final debayered image. From what I see it's not much different than for debayered one. 3:1 DNG vs. lossless one on RAW image has 54dB PSNR.


As far as the final image goes, there's really not going to be any difference as long as the codec is good, which is almost redundant with "modern" these days. Space wise there is some additional savings from being a single file instead of a collection, both from a file system allocation point of view (fewer wasted blocks due to an image exceeding a block but just barely), and only one copy of the Bayer pattern bit map for an entire clip rather than one for every frame as is the case for cDNG.
Rakesh Malik
Cinematographer, photographer, adventurer, martial artist
http://WinterLight.studio
System:
HP Spectre x360 Kaby Lake-R w/ 1080Ti eGPU in RazerCore
Offline

Andrew Kolakowski

  • Posts: 3780
  • Joined: Tue Sep 11, 2012 10:20 am

Re: ProRes RAW for UMP

PostFri Apr 27, 2018 6:18 pm

Last message had no much reference to ProRes RAW itself. It's about RAW vs. debayered image- if they are any different in nature/need different coded design etc.
Saved bits on image sequence vs. MOV is really meaningless.
RAW metadata will be probably repeated on ProRes RAW frame headers levels (not only MOV headers) in case of trimming needs etc. It's tiny compared to actual video data.
Previous

Return to Cinematography

Who is online

Users browsing this forum: Baidu [Spider] and 14 guests