Practicality, Probability, Liklihood of Compressed Raw?

Let me start by saying, for the price, I wouldn't mind if the BMCC shot in ProRESHQ and nothing else. I have limited experience using R3Draw and even less experience with it in POST, but one thing that I noticed off the bat was just the nature of file sizes.

I guess it comes off as a two part question:

1. How are people handling the CinemaDNG file sizes, and coping with storage in redundancy? This is looking rather pricey, to a point where after projects are done, I likely wont be able to afford keeping stock capture. Feedback would be greatly appreciated!

2. Is there a possibility for BMD to create a proprietary RAW codec? While some films like Prometheus and Girl With the Dragon Tattoo were shot with RED 5:1 comp, they're also shooting in 4-4.5k which will give greater latitude in post. Just curious.

And a thousand apologies if the question was asked before. I realize the topic is a bit sensitive, given Red's recent lawsuit against Sony on the very issue. But I realize that ARRI have their own proprietary RAW codec (not sure if its licensed by Red?) and I do realize that ARRI's manpower eclipses BMD by far and away. But even comp at 2:1 would be huge.(though likely impractical)

Thanks so much!

Storage is relatively inexpensive. A 3TB 7200rpm SATA-3 bare hard drive now sells for about $130. And the cost of storage continues to fall.

Given all the possible things BMD engineers might be concerned about, should the storage requirements of uncompressed 2.5K RAW CinemaDNG files be a high priority for the company?

Cheers.

There is a compressed standard for cinemadng in the works, however, having a compressed standard and supporting it are two different things.

Thanks for your response Peter,

As I mentioned in the OP, I realize that there are certainly bigger things on BMD's plate and that companies that have managed to create their own proprietary compression systems have a significantly larger manpower than BMD.

However, I am citing it as a curiosity. May I ask Peter, what type of video work you do? I do understand that the price of storage is dropping but, even at the bottom as you quoted, storage still isn't cheap. For some of us working 70-90% of the year, and with possible dailies ranging from 1-4TB. Archiving stock footage is still expensive, and that's not considering the paranoid ones who always store in redundancy.

Don't get me wrong here, it's way better than many alternatives out there. No question, and of course, there is always transcoding to Prores 444. Many options and alternatives, but even for the sake of field work, a 30min limit per SSD (with no way to erase in camera no less) kind of hurts. It's still way better than Red's 4k (and Lord knows the Dragon's 6k) on the Redmags, but I'd imagine that a RAW codec would significantly increase the BMCC's viability in the field.

Or maybe I'm one of the paranoid ones?

There is a compressed standard for cinemadng in the works, however, having a compressed standard and supporting it are two different things.

This is interesting. I'm a big supporter of Open Source ware, and I feel like the BMCC, Bolex 16, and even maybe the KineRAW will help CinemaDNG run the vein of other collaborative open source projects that can help establish as "popularized" standard and therefore uniform support.

I know its silly of me to request it during the infancy of all this, but it is a nagging question that I hope gets answered sometime in the future.

The point of my previous post was to assert that storage is already inexpensive.

Again, 3TB of near-line storage (7200rpm SATA-3 bare HDD) costs about $130 and holds several hours of BMCC RAW footage. By comparison, an hour's worth of HDCAM-SR tape costs about $100.

Storage isn't free, but relatively speaking, it's no longer expensive.

Cheers.

... a 30min limit per SSD ...

BMCC RAW consumes approx. 256GB per 30-min. (about 5MB/frame). Likewise, a 500GB SSD can hold about 60-min. of BMCC RAW.

BMCC ProRes & DNxHD consume media & storage at about 1/5th the rate of BMCC RAW.

I thought we wanted the BMCC so we can get AWAY from Compressed formats.

I thought we wanted the BMCC so we can get AWAY from Compressed formats.

We're trying to get away from LOSSY compressed formats. Not all compression has to be lossy.

It might be that CinemaDNG already includes some lossless compression, so we might already be having our cake and eating it, too... at least for those of us who have received our BMCCs, of whom I am not (yet) one.

I thought we wanted the BMCC so we can get AWAY from Compressed formats.

We're trying to get away from LOSSY compressed formats. Not all compression has to be lossy.

It might be that CinemaDNG already includes some lossless compression, so we might already be having our cake and eating it, too... at least for those of us who have received our BMCCs, of whom I am not (yet) one.

CinemaDNG is as Raw as "open Source" Raw Gets

But Compressed formats that retain "RAW Quality" (R3D) and ease of edit are never something we would want to ignore or shy away from, And prices for storage drives may be dropping in price, But Just remember the life of a typical HD is not very good,
they need to be plugged in and used often to keep them from data loss.

I.E. You store/shelve away your HD for 4 years and it may just have lost your data the next time you try and connect it.

Hard drives usually last about 5-6 years

The conventional wisdom is that the general life span of a hard drive is approximately 5 years, but with constantly falling prices in hard drives, it is more sensible to replace your hard drives every 3 years.

If you can afford it Backup/Archive to LTO Tape,
LTO version 6 released in 2012 can hold 2.5 TB .

http://en.wikipedia.org/wiki/Linear_Tape-Open

Not sure how old these prices are, but have a read at this website,
It will help you find a Local LTO House and give you an idea of pricing per GB

http://www.cintrexav.com/lto-tape.aspx

I thought we wanted the BMCC so we can get AWAY from Compressed formats.

We're trying to get away from LOSSY compressed formats. Not all compression has to be lossy.

It might be that CinemaDNG already includes some lossless compression, so we might already be having our cake and eating it, too... at least for those of us who have received our BMCCs, of whom I am not (yet) one.

Right. Red and ARRI claim that their compressions are "virtually loss-less" meaning that image info that is lost is so small that it is not possible to detect them within the scopes of human vision.

From what a lot of people are saying 8:1, 5:1, and even 3:1 is a real sweet spot in the RED side of the world. Good enough for major motion pictures.

Darryl addressed what was my biggest concern which is REDUNDANCY. Film is an excellent storage medium, unfortunately hard disks are not. Not even SSD's have been able to solve this problem as storage in SSD is still expensive and not yet rated at long storage times.

A 3-6 year lifespan with a pro videographer/DoP stocking on average 150-250TB of data per year is BY NO MEASURE cheap. Call me a penny pincher, but this adds up. Fast.

The cost of shooting raw exists in two main places:
1. Cost of grading
2. Cost of storage

Depending on the project, these costs might seem high or low. Because BMCC is modestly priced, it attracts a huge proportion of people who are not paying much for what is in front of the camera, and the overall project is really on-the-cheap, so it may not pencil out or suit their sensibilities to shoot raw.

In those cases, just shoot ProRes or DnXHD and don't grade. I estimate a huge percentage of folks will do just that. You will not get as much value from this marvelous camera, but you will serve your clients.

Also bear in mind that single-chip 'raw' does not have R, G and B at each pixel position like a 3-chip camera does. So it is already compressed quite a bit. That is why they set the resolution to be about 20% above 1080 P or 2K. It is so that when you debayer down, you will get a nice sharp true 1080p.

The rest of us are getting big RAID systems and learning Resolve and launching ourselves into a higher orbit than before.

Also bear in mind that single-chip 'raw' does not have R, G and B at each pixel position like a 3-chip camera does. So it is already compressed quite a bit. That is why they set the resolution to be about 20% above 1080 P or 2K. It is so that when you debayer down, you will get a nice sharp true 1080p.

This is why this camera is so exciting to everyone. I really believe its a game changer. 2.5k is so obtuse, so everyone will down sample to 2k. This quality (I don't see 2.5k being a cinematic standard at all) is exactly why the BMCC is such an excellent candidate for virtually lossless RAW codec.

I know, I know, some people think this is asking for the moon. But I, and I'm sure many others, feel that a proprietary, or standardized RAW codec is a very much a bit part of the RAW movement.

Red saw it. Arri saw it. So did Sony, which is the main reason why they're being sued.

Please don't get me wrong, I'm not running in guns ablazin' and demanding it. I'm just saying, "I hope its in the works/plans/future."

Before I go on a rant all my clients buy hard drives for the work I shoot, It's In my contract that they supply backup/Archival HD's, I also keep/retain 2 copies of the footage for up to 1 year for an extra fee, but for the most part this expense falls on the client.

with that said, My contract will change now that I own the BMCC, and need to archive RAW, But I'm not sure how much money to charge or how long I will store the project.

Now my rant!

To be blunt yet as sincere as I can about this subject...Nothing lasts forever, And I doubt anyone who will visit this site will ever make a blockbuster movie that the world admires so dearly that we would want to archive it for a thousand years, Not even "Star Wars" can survive that long (Sorry J.J. Abrams)

But If you love your own work that much, and want your "Great Grandchildren" to view it in it's Raw form just as you did, and in Unprocessed form, as you imagined it and as you Shot it...Then you had better call NASA because as far as I know there is no such Digital Media Archiving that will last 150-200 years known to man!
As much as we love Digital, It has a shelf life shorter than Film.
If Film is stored in a climate controlled environment, it can last a long time, And Film can be re-scanned and restored/digitized, But once Digital (Magnetic) media is damaged or shelved for a long time it's gone for ever.

I dont really understand the discussion. Why is there always this tendency to work with tools whose requirements you cannot handle, technically, financially etc.

Since the rise of HD video people always thought now things will get so much cheaper, without spending a thought on storage and monitoring requirements.

This always baffles me.

tapatanexus

(I don't see 2.5k being a cinematic standard at all)

I hope I'm not misquoting you, I'm tired...
anyway 2.5k is exactly what we need, no more, no less for Indie Films.

This is the perfect resolution, and at $3000.00 for anyone wanting to get there Indie to the big screen!

(I don't see 2.5k being a cinematic standard at all)

I hope I'm not misquoting you, I'm tired...
anyway 2.5k is exactly what we need, no more, no less for Indie Films.

This is the perfect resolution, and at $3000.00 for anyone wanting to get there Indie to the big screen!

Misquoted a little bit. I was referring to people who were thrown off with the initial tech spec that the BMCC would shoot 2.5 and not a more cinematic standard like 2k or 4k.

I was saying while people were initially thrown off by the larger resolution, I said that it was wonderfully candidate for a compressed codec because many will choose to down sample to 2k. The extra image information at 2.5k is awesome!

Personally I'll be ingesting off SSDs to Cineform.

It's not that long ago we had to injest everything of tape, my plan is treat SSDs in the same way.

Of course I don't actually have a camera yet so my workflow is all theory.

I dont really understand the discussion. Why is there always this tendency to work with tools whose requirements you cannot handle, technically, financially etc.

I suppose it's easier to be a critic then to defend a position, but allow me to explain.

It is a little strange that a $3,000 camera has a more expensive storage solution than a $19,000 camera+.

It's like having a $15,000 car with supercar specs, but requiring 115 Leaded fuel at 5mpg.

While that illustration is hardly demonstrative of the issue at hand, it does make a point.

As I've mentioned earlier, there's no mention of anyone in this discussion of "being unable to handle tools on a technical or financial" level. We can afford it. We can handle it. Some of us are penny pinchers. Does it make it wrong? Subjective, I'd say.

Besides the obvious, there are also the practical ones in the field. There are people who are against more shots per SSD? There are also the notion of diminishing returns. Regarding compression/video loss.


Like I said, the OP I made was, is it possible and I hope there to be. This is not a thread demanding anything of BMD, or faulting them from what a consumer deems to be a "glaring" oversight. Context provides the intentions here.

This $3000 camera is a tool, It is not BMD fault that HD storage and Media storage prices have not come down, But If anyone remembers how much 1GB of Ram was going for 10 years ago knows of what I speak. The technology is not growing as fast as it was 6 to 10 years ago, But We all know
That when you purchase the fastest Computer you can afford...6 or 12 months later it's obsolete.

I have a feeling BMD started making this camera with the forethought of a "Low Cost Cinema Camera" for the average consumer.

We shall see if that "forethought" pays off, not only for them, but for all of us who will be forever bombarded with "Cinema Quality" Vimeo BMCC test footage

2. Is there a possibility for BMD to create a proprietary RAW codec? While some films like Prometheus and Girl With the Dragon Tattoo were shot with RED 5:1 comp, they're also shooting in 4-4.5k which will give greater latitude in post. Just curious.

Why would they create their own Codec ? There's nothing wrong with ProRes. It's bloody good. Very bloody good. And it's built into the camera.

No other RAW camera shoots Uncompressed and ProRes and DNx all internally. Even Alexa is an external recorder for RAW.

ProRes is damn fine. I don't understand why there's this drive to have compressed RAW. It makes bugger all difference most of the time. Even when WB isn't spot on, there is so much room in the ProRes to move things around. And unlike any other camera on the market, you can flick a menu option and get uncompressed RAW. On the same damn disk. Hello..! This is a GIGANTIC leap over anything else on the market......period.

So you have RAW for the times you need it. And it's UNCOMPRESSED RAW. And you have two of the best edit ready codecs around...all built into the camera and a zillion times better than most other compression codecs out there.

Hard Drives are cheap in the scheme of things. If you don't have enough to pay for them then shoot ProRes and get 90% of the same result that's still 150% better than the crappy 8 bit codec you're used to.

Compressed RAW would still probably need a transcode to another more edit friendly codec.

Now, it is true that the EPIC has greater resolution. Is that greater latitude in post ? For VFX ? Sure. For resolution ? Yes for sure. But in other ways ? Compression with RED is proprietary. They write the little API that goes into even third party products like Resolve so you don't REALLY know what's going on with the image processing under the hood. For many years there was endless debate as to if the RED was 12 or 16 bit. I Still don't think even RED have acknowledged this.

So even though they make RedCode free to use, it's controlled by RED in that the way the images are processed are fundamentally locked up into only being the RED way, even when you use other applications that aren't made by RED.

DNG on the other hand, is an open standard. one of the cool things with the DNG files floating around is seeing all the ways you can process these images. Anyone can see at the most very fundamental level what is happening with the image (if you have that kind of predilection). You could write an application that processes BMCC footage without needing to know anything at all or get anything from BMD.

Uncompressed is a BIG DEAL, even at a lowly 2.5K.

RAW is a big deal. But it's also most of the time, only a little bit better than ProRes.

Even if you can't cope with these formats in your workflow then you have the VERY EXCELLENT compressed formats already already built into the camera.

And a thousand apologies if the question was asked before. I realize the topic is a bit sensitive, given Red's recent lawsuit against Sony on the very issue. But I realize that ARRI have their own proprietary RAW codec (not sure if its licensed by Red?) and I do realize that ARRI's manpower eclipses BMD by far and away. But even comp at 2:1 would be huge.(though likely impractical)

Thanks so much!

RAW is not owned by RED. As far as I can tell from reading the patent, RED have claimed that they have a unique way of transforming the bayer sensor data into YUV (instead of RBG) before it is then compressed. They seem to think Sony are doing the same thing.

DNG and RAW are generic and open terms. Not at all owned by RED.

BMD have ALWAYS been about uncompressed. their very first products were video capture cards that brought uncompressed SD capture to the masses.

It's the same with camera. BMD hate compression. It's their raison d'etre. It's at the very core of this product.

Look at the pictures coming out of this camera and understand where they are coming from. They want it as pure and unadulterated as possible.

jb

I started to post a reply, Not as a rebuttal, But more or less an acknowledgment of Johns post about Raw, then I realized this is a post that needs nothing added, and nothing taken away.

Nice John,
Thank You!

John, thanks for your, I appreciate your opinion, and they're practically all congruent with what I have already stated throughout the thread.

Just to clarify, I did not intend to suggest the concept of RAW was engineered by RED or ARRI, but rather their proprietary methods of compressing raw files (R3DRAW/Redcode and ArriRAW respectively) has been big parts of their success.

Perhaps the thread title is incendiary of sorts? I don't understand the amount of negative gathering regarding the thread. As I have stated many times before, this is not a "BMD should do this" or "this omission is a gross oversight" kind of thread. I explicitly stated at the beginning of the thread that I loved the BMCC and its premise, even if it came only in ProRES. What I did discuss was the feasibility and possibility of implementing a RAW compression system. That's all.

What the thread is simply inquiring is that given the success that other companies have had with virtually lossless compression codecs, would it be possible to see it in the future, or if its in BMD's plans.

The point that simply shooting in ProRES is the already available alternative. Which I agree to! It's great! But I'm simply stating that given the proven performance of compressed RAW codecs, wouldnt it be wonderful to see a similar feature on the BMCC?

2.5k (that the Prores can't do) and at a fraction of the size comparable to that of Prores, with an image quality that is indistinguishable to the human eye. That possibility seems pretty awesome to me. Compressed RAW codecs is also another reason why they're so NLE friendly. I'm surprised there is resistance to this idea.

Regarding ProRes: While i agree, that it's a good codec, i would think it is problematic to depend the future support of your device as a hardware vendor on the arbitrariness of a company like Apple (or Avid for that matter)

If you look at the recent steps they (Apple) have been doing regarding Products like FinalCut Pro (and Final Cut X) or from one moment to the other stopping the MacPro sales in Europe etc, i would have serious doubts on how much they care about maintaining ProRes in the future.

If at some point they decide that any "Pro" Series of Products is not creating enough revenue for them (which is mainly created by the iPod and iPhone, where Prores does not really matter), they might drop it.
At least, for Avid, AV related products are their "Core" business.

I wonder when Arri, BMD and some others sit down and build their own, open source codec. Alternatively Apple or Avid could open the source for their codecs, but i doubt that this will hapen.

Regarding this, i wonder if the the BMD products use DSP's specifically encoding to these variety of codecs or if the theoretically could implement ANY kind of Codec using FPGA's - at least Atomos seems to be doing this with the Ninja etc.

I just checked out Adobe's cinema DNG page. According to that, CinemaDNG already supports lossless compression.

Now that the BMCC and KineRAW cams are released, and the Digital Bolex in the works.. (plus unknown future cams) I wouldn't be surprised if NLE's such as FCPX and Premiere Pro support cinemaDNG natively in the near future.

Including Apertus, in the open source RAW cinema camera list

With GPU's getting to be powerful enough, it will only get easier for the usual suspects to support raw video. It probably won't be that long until it becomes affordable to put a Kepler cluster on a Thunderbbolt device, for that matter.

That's the hope! It's so cool to see the D. Bolex, KineMINI coming to the CinemaDNG market. It can only mean better things (hopefully) in the future!

Now that the BMCC and KineRAW cams are released, and the Digital Bolex in the works.. (plus unknown future cams) I wouldn't be surprised if NLE's such as FCPX and Premiere Pro support cinemaDNG natively in the near future.

There is already a petition over at Adobe to start supporting CinemaDNG again, they had drop support for it a few years back, But now with the BMCC there is a new surge of interest and Adobe better get on the ball with it soon and start supporting it again.


Cheers

As it turns out, Adobe didn't ever stop supporting Cinema DNG... they just weren't supporting it directly in editing because they weren't able to get the Cinema DNG plugin to work well enough. They were as surprised as everyone else with Black Magic blindsided the industry with the Cinema Camera, too

Without GPUs reaching their current lofty heights of computing power, I don't think that it would be feasible even today for anyone to edit CinemaDNG without proxies, so that's yet another reason that BMD's timing with the Cinema Camera was excellent.