RAW vs Compressed

Hi guys,

on Tuesday I''m starting a long 11 days shoot with my BMCC. We are going to shoot a short film on it. I've done some tests with the camera shooting RAW and DnxHD Film. After applying a lut in Resolve and exporting to Quicktime, the difference is barely noticeable. Raw files are huge and after each shoot I am going to create dailies witch will take a lot of time and storage space. I am thinking of shooting compressed and using raw just for very high contrast and complicated shots. What do you suggest? Share some experience or thoughts...

We've only just got our camera and haven't had a chance to use it yet, but I can say from the outset that we won't be using the RAW option very often, if ever. From what I've seen and read it seems like RAW only really gets a chance to shine when you find yourself needing to push your vision hard in a given direction, i.e. recovering detail from very bright or very dark areas, extreme colour-grading, VFX etc. Even then, the Prores footage I've seen suggests that you can get pretty great results in all of those areas without switching to RAW.

I suspect it'll be mainly serious filmmakers who find a lot of benefits in RAW.

Yea thats my basic understanding, too. I'm not a Resolve master by any means and I doubt that for this particular project i'll be able to benefit from the raw that much. Maybe for something like commercial or music video it's fine, but for a big project I'm not so sure... What do you think guys? Every advice is going to be highly appreciated.

IMHO it's not about the difference how much you can push the footy in CC before it starts falling apart.

1. Difference in frame size: 2.5k vs HD

2. Difference in quality even if you only need HD: Footage properly down sampled to 1080 in software looks better than done in camera and has a better color resolution, because of the subsampling. Debayering is also better and you have more options.

3. Difference between compressed and uncompressed - to me, the difference is huge. Especially cause I prefer to keep it uncompressed in my complete workflow till delivery.

I always was a evangelist for uncompressed recording and workflow - since back in the SD days, when the first BM Decklink cards came out. Compared to those days (when 1 TB of disk space was about the price of my truck) storage is dirt cheap - so today, I couldn't care less about storage space.

It is of course true that uncompressed is better quality than uncompressed, and oversampling also gives advantages. But whether the improvements are justified is down to the requirements of each individual project.

Test the workflow you plan to use for a given project, and see whether the difference gives you something that is of benefit to you, compared to the additional data transfer times, storage space and transcode times required by raw.

In terms of pushing the image around in the grade, you may like to look at the demo video of my LUT plugin to see how much a compressed image in film (log) mode can be pushed around in a raw type way with the right tools. You can watch the video at http://bit.ly/12tUSs2

Compared to those days (when 1 TB of disk space was about the price of my truck) storage is dirt cheap - so today, I couldn't care less about storage space.

Well, I guess in the meantime the value of your truck might be the same as a TB-harddisk nowadays.

Thank Nick! Wonderful job! Are going to create plugins for Premiere some day?

Compared to those days (when 1 TB of disk space was about the price of my truck) storage is dirt cheap - so today, I couldn't care less about storage space.

Well, I guess in the meantime the value of your truck might be the same as a TB-harddisk nowadays.

Haha, true story!

If you are working in long form, you have to face up to some pretty stiff storage requirements to shoot in raw. I am sure you can do the math. I personally saw that at the very beginning, and invested in a mondo edit box, and I am still investing in big SSD's and hefty arrays. It is not trivial.

Yet I am on Frank's side of this issue. To me, you will give away half of what the BMCC has to offer if you shoot ProRes or DNxHD. At first you think, well, only in special situations do I really need all of those stops of DR, and only if I am shooting plates and such. You would be surprised how often you like that extra DR. Plus you have to decide what level you are trying to reach in your cinematography.

You have in your hands a true cinema camera that literally rivals the Alexa (and stores RAW to SSD, which the Alexa does NOT do -- you have to buy extra hardware for it). But to get that quality out and really appreciate it, you have to buy into the RAW workflow and learn at least some basics of Resolve.

I know most folks getting this camera will dumb it down and not get the full range out of it, for money reasons. Those same folks will tend to light cheaply or not at all, of course. But at least take a hard look at the options.

A 3TB HDD holds 4-5hrs of uncompressed 2.5k raw and comes for about §120 - doesn't sound like huge storage cost to me.

If you shoot a narrative (depending on your shooting ratio), you can get away with one or two of those, plus the same amount for backup - cheaper than SR-Tapes.

Thanks for the input guys, you are really helpful. My concern is not the storage, but the mere fact that I am a newbie in Resolve and the raw world and work flow. In my tests the difference between RAW and Compressed Film were barely noticeable. And also by viewing other peoples work and tests these seems to be the story. Probably if you are pro colorist you can be able to really make the RAW footage shine. I understand in theory why RAW is so much better, I just want to see it in real world example.
Can I offer a challenge to all of you guys who are Resolve masters, show us why RAW is so much better, because let's not forget the files are 5 times bigger.

IMHO it's not about the difference how much you can push the footy in CC before it starts falling apart.



3. Difference between compressed and uncompressed - to me, the difference is huge. Especially cause I prefer to keep it uncompressed in my complete workflow till delivery.

I always was a evangelist for uncompressed recording and workflow - since back in the SD days, when the first BM Decklink cards came out. Compared to those days (when 1 TB of disk space was about the price of my truck) storage is dirt cheap - so today, I couldn't care less about storage space.

Frank: Would appreciate a detail description of your "workflow" maintaining uncompressed until final deliverables. Having some issues sorting this out myself.
(and I agree shooting raw has so many positives once you get a handle on Resolve or the other grading apps).

I am always curious as to why the files need to stay uncompressed all the way through to master and delivery? It is very easy to go back to the uncompressed original files when needed in the workflow such as VFX and final conform and color correction. Why one needs to do creative editorial with uncompressed high data rate camera originals is questionable. The fluidity of the editorial process can be compromised unless on the beefiest of systems - ability to multicam and multilayer effects (previz) is reduced; all affecting the creative flow of the creative process. It is also common to grab a bunch of scene and go on set with a laptop and a USB/FireWire drive (perhaps Thunderbolt if on Mac) to work as needed. On a small form factor 2TB drive, the difference between 5 hours of uncompressed versus 130 hours of DNxHD 36 (or ProRes proxy, for example).

Moving and archiving frame based media also takes a long time compared to clip based - especially with an archive to LTO - The tape drive can not sustain a long transfer rate as it can for larger single clip based media.

I certainly see the value of removing the conform process from the pipeline should one work at the final mastering quality, but at this point in time, the conform process does not add a whole lot to the process for the gains and price reduction for systems used in editorial. This is something that is happening now in some television shows where they shoot ProRes or DNxHD and edit with that so the final conform is basically a consolidate. But shared storage requirements, throughout and turnaround are not quite there yet for uncompressed DNG, ArriRAW, DPX, or other form of uncompressed imaging format. Although a different conversation altogether; reality shows shooting 4000 - 5000 hours for a series at a compressed MPEG codec still transcode to SD in order to keep everything on storage during editorial. It's all relative to the program type, postproduction needs, etc.

If not in a collaborative environment, work on a single system, all can be done in one app or easily between apps on the same system - then go for it!


Michael

Can I offer a challenge to all of you guys who are Resolve masters, show us why RAW is so much better, because let's not forget the files are 5 times bigger.

There's not much reason for that challenge, given how obvious the differences are, both from the point of view of image quality due to a higher bit rate and better de-Bayer process and from the higher resolution that allows you more options for re-framing in post.

The question comes down to your workflow.

Having said all that, for commercial work I'll most likely stick to ProRes for most of my shooting (once my BMCC arrives, that is), because for small projects with rapid turnaround, there isn't much time to exploit the benefits of RAW. Plus, even in ProRes, the difference in image quality between what you can get from the BMCC and what you can get from a dSLR is enormous.

Whenever I can, I'll use RAW though -- especially for the environmental documentary I'm going to be working on this summer; for the epic photography I'll be able to use in that, RAW is the way to go.

Frank: Would appreciate a detail description of your "workflow" maintaining uncompressed until final deliverables.

It all depends on the project, time, budget, customer of course.
But here is what I do, if I have the choice:

Record raw - run a LUT over it in Resolve and render out proxies - edit with those proxies in Premiere - After picture lock, I send a XML to Resolve - do my color - render out a uncompressed fullrez master (mostly DPX).

Depending on, if I need titles, VFX or something else special, that may vary a bit.
When all is said and done, I output to delivery formats as needed (TV, DCP, WEB), from that master file.

My alternative workflow is, transcoding everything to CineformRaw on the fly, while dumping the material. Since Cineform edits like DnxHD/ProRes, it's a no brainer, once it is transcoded, but needs less storage and you keep the 2.5k and all the raw benefit.

I think the benefits from RAW are most apparent when you aren't working on a proper set with controlled lighting. If you working on a set, and are able to light the scene exactly how you want it to look on your monitor, you won't really need the flexibility of RAW. However, if you are trying to get a consistent look, across many shots with various lighting conditions, the flexibility of RAW becomes tremendously valuable.

In other words, when you don't have access to a fill light, and/or the proper filter, RAW becomes your fill light and your filter.

10 bit DnxHD/ProRes is flexible enough to push it around without much damage (as long as you don't overdo it).

For me it's mostly resolution and image quality.
Subsampling 2k down to HD in software gives you real HD resolution from a Bayer pattern chip.
Also makes a lot of moire problems disappear (not all).
Doing the same thing in camera, makes quite a difference for me. But not everybody cares for that.

Can I offer a challenge to all of you guys who are Resolve masters, show us why RAW is so much better, because let's not forget the files are 5 times bigger.

There's not much reason for that challenge, given how obvious the differences are, both from the point of view of image quality due to a higher bit rate and better de-Bayer process and from the higher resolution that allows you more options for re-framing in post.

When you get the camera my friend, make some tests and you will see what I mean. Again this is just tech stuff that we are all aware of, but when I shot my tests there was obvious difference between video mode and film mode in terms of DR. But between RAW and Compressed Film, barely noticeable. In Resolve I just applied the blackmagic LUT and my expectations where to see bigger difference in DR, but there was none. This is why I challenge the more experienced guys, to show us how really RAW takes it to other level. So far, only spec talk, but no real examples!
Cheers!

I think the benefits from RAW are most apparent when you aren't working on a proper set with controlled lighting. If you working on a set, and are able to light the scene exactly how you want it to look on your monitor, you won't really need the flexibility of RAW. However, if you are trying to get a consistent look, across many shots with various lighting conditions, the flexibility of RAW becomes tremendously valuable.

In other words, when you don't have access to a fill light, and/or the proper filter, RAW becomes your fill light and your filter.

absolutely true!

IMHO it's not about the difference how much you can push the footy in CC before it starts falling apart.

1. Difference in frame size: 2.5k vs HD

2. Difference in quality even if you only need HD: Footage properly down sampled to 1080 in software looks better than done in camera and has a better color resolution, because of the subsampling. Debayering is also better and you have more options.

3. Difference between compressed and uncompressed - to me, the difference is huge. Especially cause I prefer to keep it uncompressed in my complete workflow till delivery.

I always was a evangelist for uncompressed recording and workflow - since back in the SD days, when the first BM Decklink cards came out. Compared to those days (when 1 TB of disk space was about the price of my truck) storage is dirt cheap - so today, I couldn't care less about storage space.

+1 for you sire...

Frank: Would appreciate a detail description of your "workflow" maintaining uncompressed until final deliverables.

It all depends on the project, time, budget, customer of course.
But here is what I do, if I have the choice:

Record raw - run a LUT over it in Resolve and render out proxies - edit with those proxies in Premiere - After picture lock, I send a XML to Resolve - do my color - render out a uncompressed fullrez master (mostly DPX).

Depending on, if I need titles, VFX or something else special, that may vary a bit.
When all is said and done, I output to delivery formats as needed (TV, DCP, WEB), from that master file.

My alternative workflow is, transcoding everything to CineformRaw on the fly, while dumping the material. Since Cineform edits like DnxHD/ProRes, it's a no brainer, once it is transcoded, but needs less storage and you keep the 2.5k and all the raw benefit.

Very useful info Frank! Couple of questions.
1.Does the format and encoding of proxies matter , because I am thinking of H.264 quicktime export from Resolve to Premiere. Is this reasonable, considering the fact that the editor doesn't have very fast system.
2. Have you experienced problems with the XML files when more complicated timeline with various tracks and transitions is present. I've heard that people are having serious issues.
3. How do you transcode Cinema DNG to Cineform?
Cheers, thank you for your input, it is really helpful.

1. Whatever works best in your editor. But on slower computers, Prores/DnxHD/Cineform proxies should be more fluid than H264.
H264 is a pretty CPU hog, but nor so demanding on disc speed. You have to try.

2. No, but since I do mostly commercials and narrative, simple hard cuts is all I do. Can't even remember when I used a dissolve the last time, besides fade to white/black. Make sure your Premiere timeline is clean and tidy, before you XML.

3. I put my SSD into a docking station. Instead of dumping the raw files on a drive, I drop them - directly from the SSD - into CineformStudio and render them out as CineformRaw to a HDD.

Only downside, audio is not embedded in the moment. I have to copy and sync by hand for now. But this may change as we speak. There is some hope for an update at NAB.

Thanks for the workflow explanation Frank.

It's weird - I am doing something wrong ... I do my primary simple single node grade > render out ProRes proxies and import to Premiere to cut. Do nothing complicated other than cuts and std. cross dissolves > make a Final Cut XML out of the Premiere Sequence.

But when I go to import this XML back into Resolve (and yes, I check the little config box telling Resolve to ignore file extensions) Resolve will not show me the original CinemaDNG raw files to point Resolve too.

I am thinking I am screwing up and changing the file names somewhere in this workflow.

Hopefully Resolve 10 is going to make this all moot and much easier.

Thanks again Frank for the help!

1. Whatever works best in your editor. But on slower computers, Prores/DnxHD/Cineform proxies should be more fluid than H264.
H264 is a pretty CPU hog, but nor so demanding on disc speed. You have to try.

2. No, but since I do mostly commercials and narrative, simple hard cuts is all I do. Can't even remember when I used a dissolve the last time, besides fade to white/black. Make sure your Premiere timeline is clean and tidy, before you XML.

3. I put my SSD into a docking station. Instead of dumping the raw files on a drive, I drop them - directly from the SSD - into CineformStudio and render them out as CineformRaw to a HDD.

Only downside, audio is not embedded in the moment. I have to copy and sync by hand for now. But this may change as we speak. There is some hope for an update at NAB.

Frank,
you are saving me man! Just a couple of questions: Should I make the proxies 2400x1350 or HD.The first option seems right , but I am not sure how Premiere handles them with slower machines. Also have you experienced problem with syncing sound when shooting with 24fps as mentioned here (number 5) http://nofilmschool.com/2013/04/12-thin ... ma-camera/. What frame rates do you use?

Thanks for the workflow explanation Frank.

It's weird - I am doing something wrong ... I do my primary simple single node grade > render out ProRes proxies and import to Premiere to cut. Do nothing complicated other than cuts and std. cross dissolves > make a Final Cut XML out of the Premiere Sequence.

But when I go to import this XML back into Resolve (and yes, I check the little config box telling Resolve to ignore file extensions) Resolve will not show me the original CinemaDNG raw files to point Resolve too.

I am thinking I am screwing up and changing the file names somewhere in this workflow.

Hopefully Resolve 10 is going to make this all moot and much easier.

Thanks again Frank for the help!

John,
when opening the xml in Resolve try to uncheck "Automatically import source clips into media pool". See if this helps.
Cheers

you are saving me man! Just a couple of questions: Should I make the proxies 2400x1350 or HD.The first option seems right , but I am not sure how Premiere handles them with slower machines. Also have you experienced problem with syncing sound when shooting with 24fps as mentioned here (number 5) http://nofilmschool.com/2013/04/12-thin ... ma-camera/. What frame rates do you use?

Size doesn't mater, they are just proxies, you could even make them SD (I usually make them HD though).

Side note: A lot of XML problems turned out to be user problems. You have to set your fps in Resolve before you import anything - that can't be changed later. You have to do it right when you create a new project. I saw some folks just clicking on the default project. later they wonder why it's all messed up. When you start with a 30fps Resolve project, edit in a 24fps Premiere project, and XML that back into a 30fps Resolve project - that's asking for trouble.

The sound Sync problem is actually a FCP7 bug. Premiere handles real 24p just fine (don't know about FCPX since I'm on PC).

Depending on the project, 24 and 25.

@John, yeah - what Milen said - unchecking "Automatically import source clips into media pool" should do the trick.

When you get the camera my friend, make some tests and you will see what I mean.

I tried it with the loaner that my vendor let me use for a little while, so my earlier comment wasn't based on specs, it was based on trying it. There's no challenge; uncompressed, 12-bit, 2.5K RAW offers a significant advantage over 10-bit, HD, compressed ProRes.

When you get the camera my friend, make some tests and you will see what I mean.

I tried it with the loaner that my vendor let me use for a little while, so my earlier comment wasn't based on specs, it was based on trying it. There's no challenge; uncompressed, 12-bit, 2.5K RAW offers a significant advantage over 10-bit, HD, compressed ProRes.

Ah, Ok my bet! Wish you faster delivery of your BMCC, so you can share some nice DNG's.

Cheers!

Ah, Ok my bet! Wish you faster delivery of your BMCC, so you can share some nice DNG's.

Cheers!

Thanks! I'm hoping to be able to share some stuff shot with it soon(ish)... I've been recruited to DP a 30-minute film and a documentary this summer, plus a web series pilot. The first one up will probably be the 30-minute one, and I'm hoping to have my BMCC in time for that.