BMCC RAW footage storage space required calculation?

The place for questions about shooting with Blackmagic Cameras.
  • Author
  • Message
Offline

Jules Bushell

  • Posts: 1026
  • Joined: Wed Sep 05, 2012 3:06 am
  • Location: London, England

BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:03 pm

Hi,

Edit: Thanks for pointing out the mistake. Each frame is indeed 5MB, I found out why.

From the wiki article on Bayer Filter (http://en.wikipedia.org/wiki/Bayer_filter)
Image

The diagram shows 6x6 = 36 pixels. I assumed RGB per pixel (= 36x3 108 elements), wrong! Elements are arranged in a bayer pattern with twice as many green elements to red or blue. This gives 18 green + 9 red + 9 green = 36 elements.

So storage per pixel is not 12 bits x 3 (or 4.5 bytes) as I thought, but 36/108 * 12 * 3 = 1.5 bytes per pixel. I.e.
2432 x 1366 x 1.5 = 4983168 or approx. 5MB per frame


So with my calcs corrected I get..

BMCC resolution of camera:
2432x1366

12 bits per channel, bayer pattern (twice as many green elements as red or blue)
1.5 bytes per pixel

24 frames per sec
= 2432 x 1366 x 1.5 x 24 = 119.596 MB per sec

For a 2 hour feature film
= 119.596 x 60 x 60 x 2 = 861.09 GB storage

Assuming shooting at 10:1 ratio
= 8.6109 TB + 0.8619 GB = 9.472 TB

Double this for redundancy
= 18.94 TB

So for a 2 hour feature film, typically need about 19 terabytes to store 2 copies of your BMCC RAW footage and final master?


Thanks for your help,
Jules
Last edited by Jules Bushell on Thu Jan 10, 2013 11:41 pm, edited 2 times in total.
Jules Bushell
url: www.nonmultiplexcinema.com
url: www.filmmeansbusiness.com
url: www.blurtheline.co.uk
Offline

Gavin Benjamin

  • Posts: 87
  • Joined: Wed Aug 22, 2012 2:31 pm
  • Location: Toronto

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:09 pm

Think im gonna need a couple more drives... :lol:
Last edited by Gavin Benjamin on Thu Jan 10, 2013 11:29 pm, edited 2 times in total.
G.
Offline

Felix Steinhardt

  • Posts: 205
  • Joined: Sat Aug 25, 2012 9:45 pm
  • Location: Karlsruhe / Germany

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:10 pm

It´s 5 MB per frame so 5 x 24 = 120 MB/sec

864 GB for 2 hours * 10/1 shooting ratio = ~8.6TB * 2 (Backup) = 17.2 TB or 30 % what you thought
Last edited by Felix Steinhardt on Thu Jan 10, 2013 11:11 pm, edited 1 time in total.
Offline

David

  • Posts: 321
  • Joined: Thu Aug 23, 2012 3:05 am
  • Location: Ohio.... just marginally better than Kalgoorlie

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:11 pm

It's 5mb per frame.

EDIT: sorry, Felix beat me to it.
David Daniel
Dilettante
Offline

Felix Steinhardt

  • Posts: 205
  • Joined: Sat Aug 25, 2012 9:45 pm
  • Location: Karlsruhe / Germany

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:13 pm

Is there any substitute for LTO drives?

And I find a 10/1 shooting ratio pretty pessimistic. I often repeat scenes, never do it in one take but I never exceeded 7:1
Offline
User avatar

Nick Bedford

  • Posts: 352
  • Joined: Wed Aug 22, 2012 3:56 am
  • Location: Brisbane, Australia

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:22 pm

You'd probably have enough and some breathing room with 10TB x 2.

Which in the end is not really that bad at all for a feature length film.
Nick Bedford, Photographer
http://www.nickbedford.com/
Offline

Jules Bushell

  • Posts: 1026
  • Joined: Wed Sep 05, 2012 3:06 am
  • Location: London, England

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:49 pm

Felix Steinhardt wrote:Is there any substitute for LTO drives?

And I find a 10/1 shooting ratio pretty pessimistic. I often repeat scenes, never do it in one take but I never exceeded 7:1


Back it up to film. I think movie studios still do this anyway? Should last 30+ years. Not sure how much it would cost nowadays.

Jules
Jules Bushell
url: www.nonmultiplexcinema.com
url: www.filmmeansbusiness.com
url: www.blurtheline.co.uk
Offline

Erik Swan

  • Posts: 53
  • Joined: Thu Jan 10, 2013 11:55 pm

Re: BMCC RAW footage storage space required calculation?

PostThu Jan 10, 2013 11:58 pm

None of you are taking into account the lossless compression on the DNG files. It would vary depending on what you're shooting but the required storage could be reduced by up to ~20%.
Offline
User avatar

Frank Glencairn

  • Posts: 1943
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: BMCC RAW footage storage space required calculation?

PostFri Jan 11, 2013 12:38 am

Just for the record - since 3TB drives (Segate Baracuda) that hold 2.5 hrs of 2,5k 12 bit raw material are around $130 and less, and HDCAMSR tape for the same amount of compressed MPEG-4, 10 bit. 1920*1080 are $230 you not only save 100 bucks per medium, but also get way better resolution and quality.

...just saying ;)
https://sites.google.com/view/frankglencairn/home
Offline
User avatar

Nick Bedford

  • Posts: 352
  • Joined: Wed Aug 22, 2012 3:56 am
  • Location: Brisbane, Australia

Re: BMCC RAW footage storage space required calculation?

PostFri Jan 11, 2013 1:19 am

I've been mucking around with MY BMCC!! and the files are pretty much all 5MB.
Nick Bedford, Photographer
http://www.nickbedford.com/
Offline

John Brawley

  • Posts: 4499
  • Joined: Tue Aug 21, 2012 7:57 am
  • Location: Los Angeles CA

Re: BMCC RAW footage storage space required calculation?

PostFri Jan 11, 2013 2:24 am

Congrats Nick !

I just did a 4 day short film shoot with the BMCC. We used 2.8TB recording to DNG.

JB.
John Brawley ACS
Cinematographer
Los Angeles
Offline
User avatar

Nick Bedford

  • Posts: 352
  • Joined: Wed Aug 22, 2012 3:56 am
  • Location: Brisbane, Australia

Re: BMCC RAW footage storage space required calculation?

PostFri Jan 11, 2013 2:40 am

John Brawley wrote:Congrats Nick !

I just did a 4 day short film shoot with the BMCC. We used 2.8TB recording to DNG.

JB.


Very cool! I've already got requests for small music video projects :)

2.8TB worth of storage with at least a backup is cheap as chips to finance.
Nick Bedford, Photographer
http://www.nickbedford.com/
Offline

Hundo Hill

  • Posts: 89
  • Joined: Mon Dec 10, 2012 2:12 am

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 7:44 pm

Hey Nick, John, or anyone else for that matter, I'm currently pricing gear and need help selecting memory.
What are some great medium to low cost recommendations for storage, out of what I've listed or otherwise?

Of course handling the DNG's is very important and I'm not very versed in speed configurations so I've listed a variation of drives with different memory specs:

http://www.bhphotovideo.com/c/product/7 ... I_Pro.html

http://www.bhphotovideo.com/c/product/8 ... _RAID.html

http://www.bhphotovideo.com/c/product/7 ... 4_BAY.html

http://www.bhphotovideo.com/c/product/8 ... Elite.html

http://www.bhphotovideo.com/c/product/8 ... _Hard.html

http://www.bhphotovideo.com/c/product/9 ... _raid.html

Or should I use something like this?

http://www.bhphotovideo.com/c/product/8 ... bo_5D.html

Again, I want to make sure the drive can handle to 2.5k files so I'm asking from a performance first perspective, then price, then overall bang for buck.

Thanks
Offline

Pavel Lavrov

  • Posts: 800
  • Joined: Wed Dec 19, 2012 5:24 am

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 8:53 pm

Right now I have LaCie 6TB Dual HDD Raid with USB 3.0, set up as raid 1 for protection. But for my next one I would definitely go for Drobo 5D.

Here's some up points aside from price:
You can mix and match between HDD and SSDs.
Raid 5 Plug and play.
UBS 3.0 + Thunderbolt
Battery to back up data when power cuts of....
hot-swap the drive if out of space (just amazing)
and all the other features...
Pavel L
Offline
User avatar

rick.lang

  • Posts: 18658
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 9:05 pm

Hundo wrote:Hey Nick, John, or anyone else for that matter, I'm currently pricing gear and need help selecting memory.
What are some great medium to low cost recommendations for storage, out of what I've listed or otherwise?

Of course handling the DNG's is very important and I'm not very versed in speed configurations so I've listed a variation of drives with different memory specs:

http://www.bhphotovideo.com/c/product/7 ... I_Pro.html

http://www.bhphotovideo.com/c/product/8 ... _RAID.html

http://www.bhphotovideo.com/c/product/7 ... 4_BAY.html

http://www.bhphotovideo.com/c/product/8 ... Elite.html

http://www.bhphotovideo.com/c/product/8 ... _Hard.html

http://www.bhphotovideo.com/c/product/9 ... _raid.html

Or should I use something like this?

http://www.bhphotovideo.com/c/product/8 ... bo_5D.html

Again, I want to make sure the drive can handle to 2.5k files so I'm asking from a performance first perspective, then price, then overall bang for buck.

Thanks


Looking for performance first is important and you certainly have covered a wide range of possible solutions with those choices. I am sure there are professionals making use of a variety of those choices for single operator or small post facilities especially true of the Pegasus as they were first to support Thunderbolt. It was interesting that your first choice included specific mention of 2K DPX as that is similar to the CinemaDNG files you want to handle. Still you can be careful to examine all the things that are important and also consider if you think you will want to expand beyond these or if these capacities (8TB bare) are adequate for the few projects you might have in progress concurrently.

Noise is also a consideration if the drives will be on a desktop or hidden in a closet so you may want one that is quiet or not care about noise at all.

Do you want your RAID to be striped or mirrored or both? Do you want hot-swappable drives or a RAID 5 or 6 configuration?

If you have Thunderbolt on your computer(s), I would not include the options that do not support Thunderbolt except to use as backup.

Some of these options haven't been around for long but look for benchmark results that include testing of video capabilities. Barefeats.com might be a good source for that but other sites like MacWorld etc may have some comparisons.

Don't trust the specs without seeing what real world use others have made that will be comparable for your needs. I know the new Drobo 5D models are interesting but it is not a certainty yet that their mSATA approach will be a benefit to your needs.

Personally I would either relax with a Cadillac (high-end 8-16TB option using a RAID configuration that included redundancy) or a Mustang (4TB 2bay striped TB) and work with the limitations of the Mustang as long as it had the raw horsepower needed to do the job, but did not include any backup capability. The tricky complication is you really need fast performance and only the Mustang may deliver it in a modest package but the Cadillac might be just fast enough and is a much safer ride when you are not relying on RAID striping to get risky performance! Faithfully backing up a Mustang is a must.

I've read quite a bit about the new Drobos and the tried and true Promise Pegasus and tend to lean to the real world performance achieved by professionals happy with the Pegasus. On paper the Drobo looks like a giant killer but how it really handles very large streams of DNG is not widely known yet. They are the Cadillacs for a modest post facility. Well Promise also serves rather large shops but not with the R4 or R6 but that's another story...

If you can wait a while, you are likely going to see benchmarks that will help you decide but keep your needs in mind and not the questionable things that benchmarks often include that are irrelevant for you. All things equal, get a quiet solution.

Rick Lang
Sent from my iPad using Tapatalk HD
Rick Lang
Offline
User avatar

David Chapman

  • Posts: 472
  • Joined: Thu Nov 08, 2012 11:05 pm
  • Location: Dallas, TX

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 11:13 pm

I'm looking to redo some of my storage setup at home, but I did just finalize our working storage setup at our design studio. At this point, I wouldn't use a device without a Thunderbolt connection. After transferring Terabytes of data back and forth, anything less and you will be waiting a day or more.

The Promise systems are great for us. We have 2 R6 models, one at 12TB and the backup of that on an 18TB. The R6 performs faster than the R4 model (more drives). With multiple drives, I would definitely do Raid5.

For the amount of data this camera is going to be consuming, I'm thinking about a single R6 Promise Pegasus (12TB) as a backup for the 8TB Thunderbolt G Raid. The 12TB of the Pegasus will only be 10TB in a Raid 5. I'm also figuring when I reach 8TB (the max of my backup G Raid), I will need to archive some projects to LTO. But, since I'm not doing a feature film, 8TB should keep me for a while.

Pegasus R6 12TB: $2,230
G-Tech 8TB Raid (Thunderbolt): $749

I may also consider another 512GB SSD drive as a working drive for current projects that I would then transfer to the 8TB G Raid once finalized. The Pegasus would be backing up both the G Raid and this other SSD drive, as well as my Macbook Pro automatically with Time Machine.

(If you opt for the 18TB Promise Pegasus as your working drive, the only real way to back that up is with a duplicate Pegasus.)

Hope some of this helps.
—David
David Chapman
Just another creative dude with a camera.
Offline
User avatar

David Chapman

  • Posts: 472
  • Joined: Thu Nov 08, 2012 11:05 pm
  • Location: Dallas, TX

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 11:16 pm

http://www.barefeats.com has done a great job comparing all kinds of drives. You may investigate a little there, but you will see the Pegasus is really only beat by complete SSD solutions.
David Chapman
Just another creative dude with a camera.
Offline
User avatar

Christian Schmeer

  • Posts: 904
  • Joined: Sat Sep 29, 2012 5:07 pm
  • Location: London, UK

Re: BMCC RAW footage storage space required calculation?

PostSat Jan 19, 2013 11:55 pm

Nick Bedford wrote:I've been mucking around with MY BMCC!! and the files are pretty much all 5MB.


Yes, same here. It's 5MB per frame provided there is enough colour information in the shot. However, if you shoot something in low light the lower amount of colour information seems to reduce the size to 2-4 MB per frame.
Christian Schmeer - DP / Colourist
www.christianschmeer.com
www.vimeo.com/christianschmeer
Offline

Jules Bushell

  • Posts: 1026
  • Joined: Wed Sep 05, 2012 3:06 am
  • Location: London, England

Re: BMCC RAW footage storage space required calculation?

PostSun Jan 20, 2013 9:18 am

davechap wrote:http://www.barefeats.com the Pegasus is really only beat by complete SSD solutions.


The best performing drive I've ever seen is the one in my machine at the moment, luckily. It's the OCZ RevoDrive3, a quad raid SSD with r/w 1GB+ per sec.

I tested it with 3x6MB (HD uncompressed, 2 reads and 1 write) x 25 fps that had to be done at the interval start/stop of each frame for previs. It was very accurate.

I only wish I had more than 240GB capacity but I tend to work in <10 minute sections anyway.

Jules
Jules Bushell
url: www.nonmultiplexcinema.com
url: www.filmmeansbusiness.com
url: www.blurtheline.co.uk
Offline

Hundo Hill

  • Posts: 89
  • Joined: Mon Dec 10, 2012 2:12 am

Re: BMCC RAW footage storage space required calculation?

PostSun Jan 20, 2013 3:37 pm

rick.lang wrote:
Personally I would either relax with a Cadillac (high-end 8-16TB option using a RAID configuration that included redundancy) or a Mustang (4TB 2bay striped TB) and work with the limitations of the Mustang as long as it had the raw horsepower needed to do the job, but did not include any backup capability. The tricky complication is you really need fast performance and only the Mustang may deliver it in a modest package but the Cadillac might be just fast enough and is a much safer ride when you are not relying on RAID striping to get risky performance! Faithfully backing up a Mustang is a must.

Rick Lang
Sent from my iPad using Tapatalk HD


Rick, would you recommend any particular Cadillac or Mustang?


davechap wrote:The Promise systems are great for us. We have 2 R6 models, one at 12TB and the backup of that on an 18TB. The R6 performs faster than the R4 model (more drives). With multiple drives, I would definitely do Raid5.

For the amount of data this camera is going to be consuming, I'm thinking about a single R6 Promise Pegasus (12TB) as a backup for the 8TB Thunderbolt G Raid. The 12TB of the Pegasus will only be 10TB in a Raid 5. I'm also figuring when I reach 8TB (the max of my backup G Raid), I will need to archive some projects to LTO. But, since I'm not doing a feature film, 8TB should keep me for a while.

Pegasus R6 12TB: $2,230
G-Tech 8TB Raid (Thunderbolt): $749

I may also consider another 512GB SSD drive as a working drive for current projects that I would then transfer to the 8TB G Raid once finalized. The Pegasus would be backing up both the G Raid and this other SSD drive, as well as my Macbook Pro automatically with Time Machine.

(If you opt for the 18TB Promise Pegasus as your working drive, the only real way to back that up is with a duplicate Pegasus.)

Hope some of this helps.
—David


David, I "virtually" have the budget for the set up you listed, Pegasus + G-Tech Raid; however, I plan on getting wide range of additional gear so I'm trying to limit my memory option to roughly around $1300-$1600.

So you're saying a possibility is to record to whatever SSD is in the cam, then transfer to a different 512GB SSD to edit with, then backup on the G Raid? Never thought of that configuration but I like that idea, if that's what you're saying. Up front I'll be shooting music videos in ProRes, I've written 2 shorts & working on a feature now (which will all be shot in 2.5k) but I'm sure this set up will hold me over until I'm actually ready for a longer shoot. Thanks for the help!

Oh, and most SSD's I've see are around 500-556 MBps, might you have a few recommendations for reliable SSD's?

Thanks again!
Offline
User avatar

Remo Pini

  • Posts: 176
  • Joined: Tue Dec 18, 2012 4:33 pm

Re: BMCC RAW footage storage space required calculation?

PostSun Jan 20, 2013 5:30 pm

I don't know about you, but I have different storage setups for working (local storage), storing and archiving (two nas raid boxes).

My assumption is that you hardly ever need to work with more then a few TB of actual footage at any point in time, but that needs to be as fast as you can make it.

Local Storage
I'm using 4x fast spinning 2TB drives in a Raid 0 setup (8 TB) for the local storage (SSDs are just not that cost efficient yet). This HAS to be backed up daily because one failed drive hoses the whole array! (I have a script that does that differentially to one of the NAS'es). I also have separate smaller drives for the various cache / temp / ingest / export directories (keeping those separate often gives you more performance benefit than getting faster drives). Since I have a PC based setup, those drives are all attached internally using a fast hardware raid controller.

Remote Storage
The NAS drives can be in another room (can't do that with TB yet) and nobody cares about the noise they generate. Those are as many 3TB drives in a Raid-6 as possible (performance isn't a huge deal here, it's mostly about redundancy). I got two NAS with 8x 3TB each, giving me around 18TB usable space per NAS (one mirrors the other one daily).

Archiving
If you go into archiving and long term storage, you're really going into "expensive" territory. The cheapest solution with 100% accuracy currently is probably to buy a new NAS once the old one is full (it's a poor mans version of disk-to-disk backup). Tape solutions in that magnitude are quite expensive and a pain to manage (regular tape changes, regular refreshes and all that jazz), using single multi-TB disks as a backup medium is dangerous (they fail too fast/often).

In my experience, drives fail and raids fail as well. Having less then at least two copies of your stuff is a very bad idea(tm), but with RAW files, this might become quite hard to do (until they come out with bigger/faster drives, of course. The current maximum size seems to be 4TB for a single drive, but those are not raid-suitable drives right now).
Director | Line Producer | https://grayeminence.ch
Offline
User avatar

rick.lang

  • Posts: 18658
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 22, 2013 4:51 am

Hundo wrote:
rick.lang wrote:
Personally I would either relax with a Cadillac (high-end 8-16TB option using a RAID configuration that included redundancy) or a Mustang (4TB 2bay striped TB) and work with the limitations of the Mustang as long as it had the raw horsepower needed to do the job, but did not include any backup capability. The tricky complication is you really need fast performance and only the Mustang may deliver it in a modest package but the Cadillac might be just fast enough and is a much safer ride when you are not relying on RAID striping to get risky performance! Faithfully backing up a Mustang is a must.

Rick Lang
Sent from my iPad using Tapatalk HD


Rick, would you recommend any particular Cadillac or Mustang?


I think the LaCie 2big 6TB or 8TB Thunderbolt would be my Mustang (all about power in a relatively economical package). The new G-tech option in that previous list of options looks encouraging but I have read generally that G-tech can be noisy and I don't want excessive noise on my desktop. If you know the new G-tech is quiet then it does sound like a good choice.

My Cadillac for video work remains the Promise Pegasus R6 or R4 depending upon budget but the hard decision is do you run this in RAID 0 for faster speed than the Mustang or RAID 10 to get some backup at a loss of half the capacity or RAID 5 for better backup but slower performance and a loss of about one-quarter the capacity? If I had an R6 and a Thunderbolt iMac and an SSD full of raw video I could tell you but alas I don't! We need someone with the BMCC and Thunderbolt and a Pegasus to tell us!

I wish Barefeats.com or Toms Hardware or Macworld would do a benchmark of the Pegasus running these various RAID options against a video heavy workload, ideally CinemaDNG or DPX (even more intense than DNG). I really don't care for these typical hardware tests that copy a 2GB file as if that was significant. 20GB data streams or 200GB would be better. I took a while to respond to your query because I wanted to review benchmarks but they are woefully lacking for our perspective.

I did also look at the Drobo site and I think they are putting their best foot forward for everywhere they look better than a Pegasus, but for our raw needs a Pegasus will be the better choice. A small mSATA NAND flash drive in the new Drobo 5D simply runs out of steam when transferring high volumes and then may be a drag for video work not a performance boost.

Without useful benchmarks, anecdotal evidence also points to the Pegasus as a professional video solution. But it is quite expensive and a LaCie 2big 6TB/8TB RAID 0 with nightly backups to slower WD drives may be perfect for the small or single operator facility. Not as safe as RAID 5 but adequate for many people.

Another (higher) cost option is using large SSDs in place of fast hard disks in a RAID. But I am not sure if it is necessary to rely on SSDs to handle the raw video and you may be too tight on storage capacity until 1TB SSDs became affordable. I recall reading an article on SSDs months ago that said they should not be used in a RAID 0 configuration but ever since then we are inundated with people saying they are doing just that and getting amazing performance. I am willing to wait until there is consensus that it is safe before risking that.

Rick Lang
Sent from my iPad using Tapatalk HD
Rick Lang
Offline
User avatar

adamroberts

  • Posts: 4538
  • Joined: Wed Aug 22, 2012 5:27 am
  • Location: England, UK

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 22, 2013 8:24 am

I have a Pegasus Promise R6 and Thunderbolt MacBook Pro 17".

It runs off the clock on the BMD SpeedTest.

What would you like tested?
Offline
User avatar

rick.lang

  • Posts: 18658
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 22, 2013 10:12 pm

adamroberts wrote:I have a Pegasus Promise R6 and Thunderbolt MacBook Pro 17".

It runs off the clock on the BMD SpeedTest.

What would you like tested?


Very generous offer, Adam. Thanks. I assume your R6 is being used in production though so not easy to accommodate these tests compared to if you had unused hard disks installed:

I am wondering at a minimum how the Pegasus R6 performs the Blackmagic Speed Test when the R6 is configured as RAID 0, 5, 10, and anything else you think would be important. Or how you would recommend the R6 be configured for video intensive work including processing CinemaDNG and DPX. What hard disks are you using in it?

What do you think? Not asking for much, eh? Seriously though if someone had a new one, this would be a good exercise to try before committing to a RAID configuration. Anything you can suggest would be very valuable insight since you have real experience. The bottom line: I am wondering if the RAID 5 can handle everything you throw at it or if it should be configured as RAID 0 or 10 to get maximum performance.

Rick Lang
Sent from my iPad using Tapatalk HD
Rick Lang
Offline
User avatar

adamroberts

  • Posts: 4538
  • Joined: Wed Aug 22, 2012 5:27 am
  • Location: England, UK

Re: BMCC RAW footage storage space required calculation?

PostWed Jan 23, 2013 1:19 pm

rick.lang wrote:
adamroberts wrote:I have a Pegasus Promise R6 and Thunderbolt MacBook Pro 17".

It runs off the clock on the BMD SpeedTest.

What would you like tested?


Very generous offer, Adam. Thanks. I assume your R6 is being used in production though so not easy to accommodate these tests compared to if you had unused hard disks installed:


It currently has data on it so can't reformat it yet. I will however be upgrading the drives in the near future (currently has 6 x 1TB Hitachi drives) so would be able to do various tests then.

It's currently in RAID5 format giving me a capacity of 5TB.

To give you some idea of the speed I've run the following Blackmagic speed tests on my MacBook Pro 17":
480GB SSD connected via USB2: Read: 33.6MB/s - Write: 25.5MB/s
480GB SSD connected via USB3 ExpressCard: Read: 120.5MB/s - Write: 111.1MB/s
Promise Pegasus R6 5TB RAID5 via Thunderbolt: Read: 478.5MB/s - Write: 508.6MB/s

Blackmagic Speed test lists everything as working except (see attached):
1080p50 12bit RGB 4:4:4
1080p59.94 12bit RGB 4:4:4

Promise_Thunderbolt.png
Promise_Thunderbolt.png (352.95 KiB) Viewed 17514 times
Offline
User avatar

rick.lang

  • Posts: 18658
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BMCC RAW footage storage space required calculation?

PostWed Jan 23, 2013 7:24 pm

adamroberts wrote:
rick.lang wrote:
adamroberts wrote:I have a Pegasus Promise R6 and Thunderbolt MacBook Pro 17".

It runs off the clock on the BMD SpeedTest.

What would you like tested?


Very generous offer, Adam. Thanks. I assume your R6 is being used in production though so not easy to accommodate these tests compared to if you had unused hard disks installed:


It currently has data on it so can't reformat it yet. I will however be upgrading the drives in the near future (currently has 6 x 1TB Hitachi drives) so would be able to do various tests then.

It's currently in RAID5 format giving me a capacity of 5TB.

To give you some idea of the speed I've run the following Blackmagic speed tests on my MacBook Pro 17":
480GB SSD connected via USB2: Read: 33.6MB/s - Write: 25.5MB/s
480GB SSD connected via USB3 ExpressCard: Read: 120.5MB/s - Write: 111.1MB/s
Promise Pegasus R6 5TB RAID5 via Thunderbolt: Read: 478.5MB/s - Write: 508.6MB/s

Blackmagic Speed test lists everything as working except (see attached):
1080p50 12bit RGB 4:4:4
1080p59.94 12bit RGB 4:4:4

Promise_Thunderbolt.png


Thanks, Adam! I have never seen a BMD speed test with Everything 100% from a real user's experience. The Promise web site can be frustrating as they list the R6 with performance up to 800 MB/sec but they don't say what and how it was tested exactly. I thought it was also RAID 5 using 7200 rpm hard disks but don't know much else. But do not know anyone getting that in real terms. Still what you have today does look good. Looking forward to RAID 0, 5, 10 comparisons in the future as you upgrade.

As an aside, Drobo has excellent detailed information on their benchmarks and include comparisons to a well known product (which to me seems to be the Pegasus although not identified explicitly). Wish Promise would take the time to do similarly detailed benchmarks and not show silly benchmark reports comparing USB 2 and Thunderbolt theoretical throughput.

Rick Lang
Sent from my iPad using Tapatalk HD
Rick Lang
Offline

Brandon Richardson

  • Posts: 107
  • Joined: Thu Oct 25, 2012 9:35 pm
  • Location: Washington D.C

Re: BMCC RAW footage storage space required calculation?

PostWed Jan 23, 2013 8:07 pm

For project archiving why dont you build a Duplication Tower with 3-5 Blu Ray Drives and burn project files to disc. Alot cheaper at the end of the day you get 10 disc per pack at 25 GB a piece 50 disc is 1.2TB is about 75 dollars three drives will run you about 50-60 bucks. Actual physical storage space the disc may become an issue if you have heavy workload. Its cheaper than archiving expensive 256+GB SSDS or multiple $130-150 3TB drives.

How long do you keep a project before trashing the footage? Or do you archive your complete catalog? If its the complete catalog like someone said then film may be an option but I dont know how cost effective that is.
Offline
User avatar

Trevor Zuck

  • Posts: 230
  • Joined: Wed Jan 23, 2013 8:37 pm
  • Location: the 515

Re: BMCC RAW footage storage space required calculation?

PostWed Jan 23, 2013 9:14 pm

Jules Bushell wrote:...So storage per pixel is not 12 bits x 3 (or 4.5 bytes) as I thought, but 36/108 * 12 * 3 = 1.5 bytes per pixel. I.e.
2432 x 1366 x 1.5 = 4983168 or approx. 5MB per frame...


or so its easier for everyone else...
8 bits = 1 byte
12 bits = 1.5 bytes (12/8 = 1.5)

Also all of the DNGs i've seen floating around are 2400 x 1350
2400 x 1350 x 1.5 = 4,860,000 bytes (looking at a sample frame it says 4,993,024 bytes.) I will attribute the discrepancy with the fact that I have opened the frames with Adobe Raw and adjusted the look. So meta data?

And as far as how much space is needed for a feature (120 min long film) the ratio is anywhere between 10 and 30:1 personally i'd say it tends to be closer to the high end.

Now you also have to be aware that you are going to be making prores of these to edit. Prores is like a 3:5 compression (based on a 2.12 GB animation Codec compared to ProRes 422 HQ equivalent of 1.23 GB.)

So lets say "bad" case scenario:
30 TB of acquired 12 bit 2.5k RAW footage (30:1 shot/used) + 18TB ProRes (30 x (3/5)) + 510 GB (120 min ProRes HQ final product) x 2 (RAID) ≈ 100 TB (after web files and miscellaneous previews). Not to mention graphics, VFX, and Audio.

To summarize... 1 feature = 100 TB or 50 TB/Hour
- TZ

Visual FX and Post Production Artist
Screenscape Studios
Offline

Cameron Baird

  • Posts: 9
  • Joined: Wed Sep 05, 2012 6:09 am

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 29, 2013 6:17 am

I have a Maxx Digital ThunderRAID and its a pretty serious storage device. Its Thunderbolt and an 8 drive enclosure so the speed is very fast. Replacing drives with enterprise class drives will also be affordable when the time comes. So far, Im stoked on it. Also, Maxx Digital has always been top notch with customer service.

Cameron
Offline
User avatar

Remo Pini

  • Posts: 176
  • Joined: Tue Dec 18, 2012 4:33 pm

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 29, 2013 8:20 am

rick.lang wrote:The Promise web site can be frustrating as they list the R6 with performance up to 800 MB/sec but they don't say what and how it was tested exactly. I thought it was also RAID 5 using 7200 rpm hard disks but don't know much else.


I'm pretty sure they used Raid0 for that (it basically adds the speeds of the discs in the array when reading and has almost zero processing requirements on the storage device).

Raid0 with 6 drives at ~ 130MB/sec = ~780MB/sec...
Raid5 with 6 drives at ~ 130MB/sec = ~500MB/sec... (Adam's number)
Director | Line Producer | https://grayeminence.ch
Offline
User avatar

Frank Glencairn

  • Posts: 1943
  • Joined: Wed Aug 22, 2012 7:07 am
  • Location: Germany

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 29, 2013 9:05 am

A few thoughts.

Archiving to BluRay is a No-Go for us since we know they don't last longer than a few years.

Regarding RAID trhoughput and bandwith: Keep in mind, that DNGs are single 5MB files and behave different in drive performance than a large chunk AVI or MOV file.

A lot of the stuff we shoot (commercials) has a half-life of less than a year, than it's so outdated that we could just throw it away. I also have a ton of Digibeta tapes archived here - some great stuff is on them, but it's almost useless, nobody buys SD material since years. So I have to decide what's important in the future.

Our strategy is like this. When I shoot something for a customer, I keep a working copy on my RAID, a backup of it, and 2 backups of the original material. All backups go on "naked" 3TB HDDs via docking station. After all is said and done, the customer gets a uncompressed master of the final product, the raw material and a encrypted ZIP file with all the project files and whatnot, on HDDs plus a backup of the same.
From that point on, we wipe it of our drives, and the customer is responsible for everything else (he has to sign that in the contract).

For our own stuff we do the same, except raw material and master is transcoded to Cineform.
https://sites.google.com/view/frankglencairn/home
Offline
User avatar

rick.lang

  • Posts: 18658
  • Joined: Wed Aug 22, 2012 5:41 pm
  • Location: Victoria BC Canada

Re: BMCC RAW footage storage space required calculation?

PostTue Jan 29, 2013 6:43 pm

remopini wrote:
rick.lang wrote:The Promise web site can be frustrating as they list the R6 with performance up to 800 MB/sec but they don't say what and how it was tested exactly. I thought it was also RAID 5 using 7200 rpm hard disks but don't know much else.


I'm pretty sure they used Raid0 for that (it basically adds the speeds of the discs in the array when reading and has almost zero processing requirements on the storage device).

Raid0 with 6 drives at ~ 130MB/sec = ~780MB/sec...
Raid5 with 6 drives at ~ 130MB/sec = ~500MB/sec... (Adam's number)


Things just got a little more interesting with the LaCie 5big 10/20TB RAID:
http://www.lacie.com/ca/products/product.htm?id=10607

That's the Canadian prices that I think are higher than the US prices (not because our dollar is lower than the US but because that's what LaCie does).

Following up on my earlier analogy of the LaCie 2big being a "Mustang" and the Promise Pegasus R4/R6 being a "Cadillac", I am not sure what to call the LaCie 5big when you configure 3 drives as RAID 0 and two drives as RAID 1 so you get great performance for your working copy and safe recovery for your backup without the complexity of a RAID 5. Maybe a "Mazda CX5 crossover" but that's a mouthful. I am liking this 5big with the mixed RAID, how about you?

Rick Lang
Sent from my iPad using Tapatalk HD
Rick Lang
Offline

Brandon Richardson

  • Posts: 107
  • Joined: Thu Oct 25, 2012 9:35 pm
  • Location: Washington D.C

Re: BMCC RAW footage storage space required calculation?

PostWed Jan 30, 2013 12:50 am

rick.lang wrote:
remopini wrote:
rick.lang wrote:The Promise web site can be frustrating as they list the R6 with performance up to 800 MB/sec but they don't say what and how it was tested exactly. I thought it was also RAID 5 using 7200 rpm hard disks but don't know much else.


I'm pretty sure they used Raid0 for that (it basically adds the speeds of the discs in the array when reading and has almost zero processing requirements on the storage device).

Raid0 with 6 drives at ~ 130MB/sec = ~780MB/sec...
Raid5 with 6 drives at ~ 130MB/sec = ~500MB/sec... (Adam's number)


Things just got a little more interesting with the LaCie 5big 10/20TB RAID:
http://www.lacie.com/ca/products/product.htm?id=10607

That's the Canadian prices that I think are higher than the US prices (not because our dollar is lower than the US but because that's what LaCie does).

Following up on my earlier analogy of the LaCie 2big being a "Mustang" and the Promise Pegasus R4/R6 being a "Cadillac", I am not sure what to call the LaCie 5big when you configure 3 drives as RAID 0 and two drives as RAID 1 so you get great performance for your working copy and safe recovery for your backup without the complexity of a RAID 5. Maybe a "Mazda CX5 crossover" but that's a mouthful. I am liking this 5big with the mixed RAID, how about you?

Rick Lang
Sent from my iPad using Tapatalk HD


Thanks for that link thunderbolt and hot swap very intriguing
Offline

Paul Abrahams

  • Posts: 62
  • Joined: Sun Sep 16, 2012 1:59 am
  • Location: Sydney Australia

Re: BMCC RAW footage storage space required calculation?

PostMon Jul 01, 2013 10:20 am

I just bought a Lacie 2big 4TB, it was preformatted in mirror and the Speed Test showed 170/170 read/write. I opened up resolve with about 30secs of raw footage. I had 6 nodes running and the frame rate hovered between 18-27fps.

imac 3.4GHz i7 - 16r - 1tb fusion drive - GTX680mx-2gb

I'm going to set it up as a RAID 0 which should be fast enough for grading.

I don't have a lot of funds and just starting on a short student 20min film project. Will need to sort out back up options pretty soon I'd say USB 3. This is all new to me but I'm learning fast... :-)
SubUrban Voodoo Productions

Return to Cinematography

Who is online

Users browsing this forum: Rojer Madruga, ShaheedMalik, STEFANvDIEST and 44 guests