Cost effective way to do post archives

I work on a number of Indie Film projects for clients. They're a combination of short and feature length forms ranging from 10 to 145 minutes long so far. I always return all the assets and drives to the client but I keep a final DRA of each project. This is because there might be adjustments or the client lost their drives. DRA includes all the files used in the timeline so having those files and being able to restore them is enough. I don't need to waste terabytes of clips that are bad takes, unused, etc.

But even with just DRA which I archived on an external backup drive, over time it gets filled up. 90% of those probably I won't be working on but you never know 36 months down the road. So I started thinking of an archival solution however most of the high-end Sony archiving systems is just too pricey but I get it - why people invest on them. You're paying for insurance.

Recently, I did an analysis of all the DRA files I have to date, especially those with feature length projects. I found that the largest DRA I have is an 80 minutes feature length that was 59Gb. Another 144 minutes feature length was 23Gb. On average, I will say the DRA's are around 40-50Gbs.

So I got to think. What if I used M-Disc to archive each project and just keep them safe? I can put up to 100Gb DRA project on one M-Disc and store it for a very very long time. I can reclaim my drive space and use it for work items. Best part, I can buy a couple of drives and that will set me back only $60. I can by a box of 100Gb M-disc for around $80. That's $140 and although that process of archiving is slow, I can tolerate it. I think that is my cost effective way to do post archiving, besides those disc can last 1000 years.

What are your thoughts on this?

I use M-Disc for that very purpose and a Disc Publisher for printing on the surface. Bear in mind though, as I'm sure you're aware it's a very slooow txfr. I usually burn 2 copies, despite what they say about the thousand years.

I use M-Disc for that very purpose and a Disc Publisher for printing on the surface. Bear in mind though, as I'm sure you're aware it's a very slooow txfr. I usually burn 2 copies, despite what they say about the thousand years.

I'm okay with the slow writes. I just do it at the end of the project and when it's time to archive the project. Good idea of making 2 copies.

Any recommendation on a good M-disc burner/player? How about media brands? I'm looking at the 100Gb M-Disc.

When exporting a Resolve project archive, are all the DRA files in the package comes from what's on the media bin or is it only the files that are used on the timeline?

Any recommendation on a good M-disc burner/player? How about media brands? I'm looking at the 100Gb M-Disc.

When exporting a Resolve project archive, are all the DRA files in the package comes from what's on the media bin or is it only the files that are used on the timeline?

I have both an external Verbatim burner (https://www.verbatim-europe.co.uk/en/pr ... ter-43890/) and an internal LG BU40N, I use only only Verbatim media. I use VirtualCD for test burns (super fast acts exactly like a Blu-Ray internal burner). It was all for Blu Ray/DVD authoring really, which I had a bit of work in during lockdown, but like you it occurred to me 100GB M-Disc makes sense for archive too.

Because I mainly work in Avid roundtripping to Resolve, I consolidate the final playout (50 frames handle usually) in Avid AAF that to Resolve as separate project and then export the .drp, Avid project and the AAF, so there is one set of media for all. If you export the .drp you can see what's in it by unzipping it - a series of xmls, including gallery, timeline etc. and all the bins. This is quite new and a work in progress and I don't do it yet for every gig, only those that are very important or they maybe re-versioning for.

Bear in mind too (though you've already said the size of files you're using) that what I'm dealing with are at maximum 2 x 45min shows per disc and only in HD UK broadcast (Other shortforms are 4K) yet we are still talking about at least 1GB per minute, so it's limited in scope. Hope this helps.

Steve, thanks. It's helpful. And yes, like you, I just archive the ones that are important projects - not a one off projects.

So I got to think. What if I used M-Disc to archive each project and just keep them safe? I can put up to 100Gb DRA project on one M-Disc and store it for a very very long time. I can reclaim my drive space and use it for work items. Best part, I can buy a couple of drives and that will set me back only $60. I can by a box of 100Gb M-disc for around $80. That's $140 and although that process of archiving is slow, I can tolerate it. I think that is my cost effective way to do post archiving, besides those disc can last 1000 years.

We have M-Disc drives, but I think it's kind of a goofy format, plus it only goes to 100GB. We've had every writable disc format since the beginning of time (or at least it feels like it), and I've seen CD-Rs, DVD-Rs, and BD-Rs all fail at different times. We do use LTO-6 for long-term backup, but in our client contract, we only guarantee 6 months of backups for any project. In real-life, we almost always save them for a much longer amount of time; we've had clients come back after 12-18 months to make a radical change (like a re-edit), and luckily we still had all the material, and we didn't charge them a storage fee. We're also pretty good about media management, and try to chop down projects so that we have every shot with 2-second handles, which we'll do for anything shot on Raw or ProRes. We should probably upgrade to LTO-8 later on in the year.

Cheap external hard drives are pretty fast and simple to use, and those will work for near-term storage. We've kind of standardized on 10TB drives over the past few years, and I think we have about two dozen of them covering that period. We try to keep our fast RAIDs cleared of older projects, and we have roughly 300TB of storage at the moment (and we're overdue to upgrade one of those).

Cheap external hard drives are pretty fast and simple to use, and those will work for near-term storage. We've kind of standardized on 10TB drives over the past few years, and I think we have about two dozen of them covering that period. We try to keep our fast RAIDs cleared of older projects, and we have roughly 300TB of storage at the moment (and we're overdue to upgrade one of those).

Archive has a different requirement than short to medium term storage Marc. It's about something that may a)survive and b) be playable in a 100 years say and M-Disc has been one of the formats considered viable by the pro archive industry. As you say it's main drawback is it's small size that once seemed quite large. LTO is just not viable for a small boutique facility, I looked into it, but SCSi and tape don't seem the way forward anymore to me.

Backup hard drives are the still the only real option for mass project storage with all the rushes and I have a series of portable raids that I'll keep around for 6 months after an edit finishes, more if the client asks. But for final edit important masters there is a place for M-Disc, for me, but as I say only good for HD long form UHD/4K short form, nothing else will fit. I considered SD cards too but who knows how long they really last/are stable too. Although there is an archive standard for them too.

We're also pretty good about media management, and try to chop down projects so that we have every shot with 2-second handles, which we'll do for anything shot on Raw or ProRes.

This is a good idea. I have not done this and should. Where can I find info on how this is done in Resolve? Can this be automated?

We have M-Disc drives, but I think it's kind of a goofy format, plus it only goes to 100GB.

I agree it only goes to 100GB and that's its limitation. I don't know about being a goofy format but considering the cost to benefit for small foundry, M-Disc may be a good saving grace to archive and reclaim space on the work drives. I am still researching this because I have some feature projects that the DRA's are over 3TB in size. Maybe chopping down will help a lot. LTO is nice but just too expensive especially I don't charge for archiving.

Archive has a different requirement than short to medium term storage Marc. It's about something that may a)survive and b) be playable in a 100 years say and M-Disc has been one of the formats considered viable by the pro archive industry. As you say it's main drawback is it's small size that once seemed quite large. LTO is just not viable for a small boutique facility, I looked into it, but SCSi and tape don't seem the way forward anymore to me.

Backup hard drives are the still the only real option for mass project storage with all the rushes and I have a series of portable raids that I'll keep around for 6 months after an edit finishes, more if the client asks. But for final edit important masters there is a place for M-Disc, for me, but as I say only good for HD long form UHD/4K short form, nothing else will fit. I considered SD cards too but who knows how long they really last/are stable too. Although there is an archive standard for them too.

I have the same opinion too. Forget SD cards, that won't last.

There was the beginnings of development on a standard archival form (Worm) of SD but it seems to have stagnated and was last heard of 5 years ago, and with woeful capacity:

https://gizmodo.com/sandisk-worm-write- ... st-5025423

It's a grave problem for most camera original media too, we need something that can preserve the original camera 'negative', for realistic archival purposes. It's now about transferring every few years to the latest tech around.

Archive has a different requirement than short to medium term storage Marc. It's about something that may a)survive and b) be playable in a 100 years say and M-Disc has been one of the formats considered viable by the pro archive industry. As you say it's main drawback is it's small size that once seemed quite large. LTO is just not viable for a small boutique facility, I looked into it, but SCSi and tape don't seem the way forward anymore to me.

Yes, we're well-aware of long-term archival problems since a lot of what I do is feature restoration (from film). SCSI hasn't been part of modern computer usage for at least 15 years that I know of; ours are all Thunderbolt 2.

These two articles from the Motion Picture Academy go into a lot of the details on what the challenges are with long-term film/TV storage, but there are still not a lot of solid solutions:

"The Digital Dilemma Part 1"
https://www.oscars.org/science-technolo ... al-dilemma

"The Digital Dilemma Part 2"
https://www.oscars.org/science-technolo ... -dilemma-2

Backup hard drives are the still the only real option for mass project storage with all the rushes and I have a series of portable raids that I'll keep around for 6 months after an edit finishes, more if the client asks. But for final edit important masters there is a place for M-Disc, for me, but as I say only good for HD long form UHD/4K short form, nothing else will fit. I considered SD cards too but who knows how long they really last/are stable too. Although there is an archive standard for them too.

What we did in the early 2000s was advise people to initially store on hard drive and LTO, and then plan to do a "data migration" to newer drives every 3-5 years. That's what all the Hollywood studios did for years and years. Things changed when Netflix and Amazon got into the production business, and elaborate schemes have been developed for long-term Cloud storage over time. You can argue that "Cloud storage is basically just somebody else's computer," but because of the number of backups they have, plus the fact that the files are replicated worldwide, there's bound to be duplicates available in Asia, Europe, North America, South America, and so on. It's not cheap, it's not fast, but it can work.

Of course, there's also the potential to do a film-out with projects and save to motion picture film via a laser recorder. Some actually go to the trouble of creating YCM separation masters -- individual B&W copies of each color record (RGB), which can be reassembled and printed to create new negatives and prints -- but the costs are pretty high.

It's a grave problem for most camera original media too, we need something that can preserve the original camera 'negative', for realistic archival purposes. It's now about transferring every few years to the latest tech around.

And that is pretty much what we tell people. There are firms you can hire -- Iron Mountain is a big one -- who has massive underground vaults and also huge buildings devoted to archival storage. They also have systems in place to copy and store data from just about any format, using Cloud storage, physical servers, and other means, and they can handle any kind of film, sound, television, tape, or file format that exists:

Iron Mountain Film & Sound Archive Services
1035 N Highland Ave.
Los Angeles, CA 90038
(800) 899-4766
https://www.imes.media/

Of course, the major studios have their own staffs and warehouses to manage all their old shows and films, but fires and staff cutbacks sometimes put these archives at risk, particularly in the last 20 years. Stuff gets lost, misfiled, and otherwise goes missing, and it takes vigilant effort to track down the missing materials and get everything reassembled again.

Of course, there's also the potential to do a film-out with projects and save to motion picture film via a laser recorder. Some actually go to the trouble of creating YCM separation masters -- individual B&W copies of each color record (RGB), which can be reassembled and printed to create new negatives and prints -- but the costs are pretty high.

Yes I'm aware of all this Marc, I'm talking way down the pecking order here of what Ellory and I are looking at, which is more of a boutique solution but thanks, very interesting links. The filmout for digital was once considered to be a very viable long term solution but it breaks one of the golden rules of archive, that of absolutely faithful 1:1 pristine recording in original state - it's an analogue only, and with the pixel race upwards it is increasingly becoming impossible for celluloid to record all of them.

Sony have an archive system based on Optical Blu ray but it has not garnered much interest as once again it is propriety, involving many disks working together and is quickly, as ever, being left behind in space. The optical disc in it's purest form is attractive since the pits and troughs can be scanned and interpreted in the absence of any machine capable of reproduction. Compatible worm technology like M-Disc shows there is way to make that media very long lasting but it fails in its outdated modest capacity. The cloud is probably pretty safe but that's by no means certain and there are rightsholder security concerns. It's crazy we can't come up with something big, secure, long lasting, that can sit in those faults safe even close to what we could do 70 years ago.

I know this is wishful thinking and I am not a hardware engineer by any stretch. Would it be possible to have M-Disc with capacities of up to 5Tb that does not have the cost of owning an LTO device?

For those using M-DISC for data archiving, what drives and data buying software would you recommend? This would be for an OS X system. I am currently looking at the Pioneer BDR-XS07UHD 6x Portable USB 3.1 Gen 1 Blu-ray Burner with M-DISC Support and would use Verbatim M DISC BDXL 100GB 4x Blu-ray Discs. Or maybe the 25GB discs as well.

Thanks.

I know this is wishful thinking and I am not a hardware engineer by any stretch. Would it be possible to have M-Disc with capacities of up to 5Tb that does not have the cost of owning an LTO device?

The Sony Optical Archive system uses 5.5TB cartridges but they contain 11 actual discs. They reckon an equally long lasting, though different, recording tech, with a projected Petabyte+ capacity eventually (21 drives). Fast txfr too because of cumulative discs. It's not cheap though and development was last heard from in 2017 (still current product though):

https://en.wikipedia.org/wiki/Optical_Disc_Archive

I think it's the eol for the single disc capacity development of Blu-Ray and who knows how long M-Discs will be sold. Popular with stills photographers and document archivists though, which is what's keeping it alive and the media you do use now should last too regardless.

For those using M-DISC for data archiving, what drives and data buying software would you recommend?

Any burning software that works with regular discs should work. Have a look at my earlier post for the burners too I use. The external Verbatim one is the best of the two.

Why don't you use any storage service including G drive for business (60$ a month for unlimited storage, use to be half of it).
After few years all those "discs" start to be pain to deal with (including LTO).

Any burning software that works with regular discs should work. Have a look at my earlier post for the burners too I use. The external Verbatim one is the best of the two.

Thanks Steve

Why don't you use any storage service including G drive for business (60$ a month for unlimited storage, use to be half of it).
After few years all those "discs" start to be pain to deal with (including LTO).

Not a realistic option for pure archive for the small outfit - for just 30 years that would be $21,600, without inflation and then you would need the unlimited . I started to use M-Disc for replication masters for Blu-Ray authoring and only really use it for a very small amount of mostly my own productions. The rest are client short term storage of 6 months at most and that's good old fashioned mirrored raids.

Yes, you need to keep your work coming and paying for those archives, otherwise it's costly.
In the same time I want to see you restoring those M-Discs in 30 years time ( wonder how easy it will be- software/hardware side wise).

Yes, you need to keep your work coming and paying for those archives, otherwise it's costly.
In the same time I want to see you restoring those M-Discs in 30 years time ( wonder how easy it will be- software/hardware side wise).

It's a valid point about software, Andrew. The hardware should be good - I've got a disc player from almost 20 years ago that's still good and it wasn't cared for. The Blu-Ray burners get little use. But a pc to play that software may be a different matter. It's not really the point for archive though, since the object is for the media to at least outlast the original archivist and as I mentioned earlier it is more than feasible to scan the pits to retrieve information in 500 years say - which can't be done with magnetic tape or hard drives. The cloud leaves the quandary of our inevitable demise too since who keeps up payments? And a lawyer with a will is more important than a vault.

Incidentally I was a heavy user of discs back in the day and over 20 years worth could fill 1 box - now since I use Kindle a lot I have a lot of bookshelf space too so I'm not to worried that my modest plans will take over the house

I think for small outfits, the M-disc is ideal. I think I have a strategy to hold my short form archives with the 100Gb M-disc, as I mentioned about the analysis I did above.

Then there are a the feature projects that I need to have an M-Disc strategy too because those Resolve archive (DRA) can go up to 3Tb in size. Have any of you did something like this?

Maybe something like this Ellory?:

https://www.burnaware.com/

It apparently can span backups across all forms of optical media. Why do you say "can go up to 3TB"? Surely a .dra can go well beyond this?

Maybe something like this Ellory?:

https://www.burnaware.com/

It apparently can span backups across all forms of optical media. Why do you say "can go up to 3TB"? Surely a .dra can go well beyond this?

I'll check out that software. Yes, DRA can go beyond 3TB but so far, for the indie features I've worked on, they have taken about 2 to 2.8TB of storage so that's why a 3TB would be good for now. Otherwise, we'll have to go to the more expensive LTO.

I know this is wishful thinking and I am not a hardware engineer by any stretch. Would it be possible to have M-Disc with capacities of up to 5Tb that does not have the cost of owning an LTO device?

No, I don't think that's possible. I'm very skeptical about recordable optical disc for longterm storage. LTO is actually pretty cheap if you look at it over a long period of time.

Yes I'm aware of all this Marc, I'm talking way down the pecking order here of what Ellory and I are looking at, which is more of a boutique solution but thanks, very interesting links. The filmout for digital was once considered to be a very viable long term solution but it breaks one of the golden rules of archive, that of absolutely faithful 1:1 pristine recording in original state - it's an analogue only, and with the pixel race upwards it is increasingly becoming impossible for celluloid to record all of them.

The pixels won't help you if in 50-60 years, none of the discs can be played back and none of the files work with modern software. We know film will work, and that's about it. The free Academy papers -- the links I posted -- go into this in detail.

Realistically, the fastest and most reasonable solution is to just buy a boat load of drives and resign yourself to copying them every 4-5 years and use checksums to verify that the files are OK. If you have (say) three complete copies of the entire film on hard drive, the chances of all the drives failing are extremely small. You figure if a 10TB drive is about $300 right now, and you were to buy 5 or 6 of them, that would pretty much hold all the raw footage for most modest-budget indie films. From my perspective, that's fairly affordable, even if you were to buy multiple sets of them and keep them in a bank vault.

Amazon Web Services (AWS) is used by several companies I deal with for large file data storage in the cloud, and it's actually not too expensive and not too time-consuming, provided you have a fast (1GB) fiber connection:

https://aws.amazon.com/products/storage/

AWS is ok if you don't plan to download files too often (deep archive). If you do then costs grow quite a lot.

LTO is relatively cheap, but also bit of a hassle. We had to restore many old tapes and found it so painful.

So I made the decision to try out M-Disc. It's too cheap to do so why not. I successfully burn a 87Gb DRA folder on a 100Gb optical disc. I do have larger DRA's, some in the 1200Gb size. Right now I have them backed up on TB drives. My next step is to see if I can find some software that will let me write 1200Gb of DRA files to 12 optical disc in sequence. If any of you are doing this and using some software to do this successfully, please let us know what software you're using. Thanks.

Please don't get me wrong. This is a small shop low budget archival solution. It is not to compete or replace LTOs. LTO's are relatively cheap on the long term but expensive to have in the onset. Also, I provide the archives to my clients free of charge and I think it needs to be able to read on any blu-ray or m-disc drives, which are very affordable.