Archival-quality codec with lossless compression

Which container format and codec are recommended for compressed lossless archival storage?

My source video is uncompressed AVI dvvideo, which I'm using ffmpeg to put in an MOV container and then edit in Resolve. My objective is to output several rendered versions, some h264/mp4 for online use, and something for long-term archival.

I'm uncomfortable using a lossy codec, and have lots of space available.

Thanks!

Hah, I was going to suggest FFV1 in a MKV container but it looks like you already jumped on the FFV1 thread... That combo seems to be gaining some traction in the AV archival community.

Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.

Lossless JPEG2000 is also only available for some hand picked resolutions in Resolve.

Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.

I'm disappointed that the best option, an open-source option, is not available directly within Resolve. I'm looking at a two-step process now, and will need considerably more disk space. :/

Post an addition to the DAVINCI 16 | Big feature request list:
viewtopic.php?f=21&t=71757

Blackmagic often implement user requests.

Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.

I'm disappointed that the best option, an open-source option, is not available directly within Resolve. I'm looking at a two-step process now, and will need considerably more disk space. :/

No idea why you need lossless archive? In most cases it simply not needed.
What is your source format? Where does it come from? Camera recording, game, CGI etc?
Intermediate archives are already very good quality and lossless one is more for a medical etc. usage.

You could also reverse that and ask why someone should go lossy if they have no issue at all handling lossless.

This is universal. You could also say that FLAC, ALAC and WavPack are pointless and everyone should just be happy with MP3, MPC, AAC, Vorbis, Opus, etc.

Some just don't want to unnecessarily lose quality if there is no point in it. Especially if the output is only used to transcode to one more lossy format.

No idea why you need lossless archive?

Audiovisual archiving organisations specify lossless formats.

Be it organisations such as the International Association of Sound and Audiovisual Archives or Fédération Internationale des Archives de Télévision (International Federation of Television Archives) and their members.

Requirements adhered to by all standards compliant audiovisual archives and archival/preservation services.

No idea why you need lossless archive? In most cases it simply not needed.
What is your source format? Where does it come from? Camera recording, game, CGI etc?
Intermediate archives are already very good quality and lossless one is more for a medical etc. usage.

My source is not the best. I'm working from 4 source formats - MiniDV, VHS, 8mm film, and Super 8 film. The majority is on MiniDV, and is transferred to my PC via Firewire as uncompressed AVI. The VHS is digitized to uncompressed AVI, and the Super 8 and 8mm are digitized using a digital camera AVCHD h264 at 1080p30.

Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

If you would deal with HD or UHD then your would quickly change your mind
Your 60TB would be not much at all.
SD is easy these days.
You may as well export 10bit uncompressed YUV out of Resolve. What is the average ffv1 gain in those sources, 3:1?
Problem with ffv1 is that it's basically unsupported by pro apps. In case of restore you will need about every time convert ffv1 with ffmpeg to something usable. V210 (10bit uncompressed YUV) is universal. Works in about every tool. You can RAR it if you want to preserve space (you should gain about the same as with ffv1).

Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.

You could also reverse that and ask why someone should go lossy if they have no issue at all handling lossless.

This is universal. You could also say that FLAC, ALAC and WavPack are pointless and everyone should just be happy with MP3, MPC, AAC, Vorbis, Opus, etc.

Some just don't want to unnecessarily lose quality if there is no point in it. Especially if the output is only used to transcode to one more lossy format.

When you digitise something (specially analog) from the tape then going to something like ProRes or uncompressed gives basically same end quality. You will never see a real difference and then your digital file becomes main master (and at this point loss of eg. 1% in quality is not important at all). For SD you can easily do lossless, but for HD or UHD this is not so easy, specially if you have 1000s of hours to backup.
For mentioned MiniDV I would simply capture it as DV and keep original DV files (although I hate DV format).

Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.

For editing, cleaning up, fixing. If you have time it's good thing to do, rather than just dumping crap into your archive. Resolve for me is very bad choice for this job, but you can still use it.

Try VirtualDub2 http://virtualdub2.com/ and pick the codec you like.

Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.

For editing, cleaning up, fixing. If you have time it's good thing to do, rather than just dumping crap into your archive. Resolve for me is very bad choice for this job, but you can still use it.

I have seen a lot of "cleaning" and "fixing" that actually destroyed information. I agree with you that Resolve is not the right tool for processing low fi interlaced material. I would recommend Vapoursynth for that.

no issues with disk space + display refered + archive = DPX to LTO
no issues with disk space + scene refered + archive = EXR to LTO

LTO takes space and costs quite big money as well

Can't we just allow people wanting a lossless workflow to do things their way and people wanting a lossy workflow to do things their way.

Resolve supports a gazillion lossy variants, I don't think someone will die if it gains support for one decent lossless codec.

Different metrics have different weight to different people with different use cases on different hardware in different situations.

When you want better watchable quality than the originals, this might be a solution https://www.redsharknews.com/production/item/6227-how-ai-is-helping-to-make-the-best-classic-tv-series-restorations for some of your source material.

Why can't we just allow people wanting a lossless workflow to do things their way and people wanting a lossy workflow to do things their way.

Resolve supports a gazillion lossy variants, I don't think someone will die if it gains support for one decent lossless codec.

Different metrics have different weight to different people with different use cases on different hardware in different situations.

No one will die. No one will protest I assume, but list of the new features "needed in Resolve is about endless.
I 10000x prefer to see ProRes export than ffv1.
Maybe there is a reason for gazillion lossy formats and none of the lossless ones

Maybe there is a reason for gazillion lossy formats and none of the lossless ones

I'm curious how that makes sense, it feels like I must be misunderstanding something here.

If I were a professional working on e.g. a movie, I'd want to retain as many artifacts as possible in the best possible quality. I'd keep all the original footage and everything I render through Resolve in a lossless compressed format. This way, in 10 years even if I cannot reproduce the render using Resolve, I still have the master render to use as an input to further edits/ improvements etc. in any tool I wish. Why would I want to start from a lossy version? It'd be like a photocopy of a photocopy.. each time losing quality and original data.

Forgive my lack of insight into the video industry, but assuming the final movie output is rendered by Resolve, wouldn't a lossless compressed master and a lossy compressed versions be required by the client, or for my own safe keeping?

What are pros delivering when say a 4k Bluray is getting pressed? Lossy codecs?

No one will die. No one will protest I assume, but list of the new features "needed in Resolve is about endless.

OK, so let's both pitch for FFMPEG. That way we can both be happy

Maybe there is a reason for gazillion lossy formats and none of the lossless ones

I'm curious how that makes sense, it feels like I must be misunderstanding something here.

If I were a professional working on e.g. a movie, I'd want to retain as many artifacts as possible in the best possible quality. I'd keep all the original footage and everything I render through Resolve in a lossless compressed format. This way, in 10 years even if I cannot reproduce the render using Resolve, I still have the master render to use as an input to further edits/ improvements etc. in any tool I wish. Why would I want to start from a lossy version? It'd be like a photocopy of a photocopy.. each time losing quality and original data.

Forgive my lack of insight into the video industry, but assuming the final movie output is rendered by Resolve, wouldn't a lossless compressed master and a lossy compressed versions be required by the client, or for my own safe keeping?

What are pros delivering when say a 4k Bluray is getting pressed? Lossy codecs?

Pro movies use DPX or EXRs so this is uncompressed (from 10 to32bit float precision). No one uses lossless as it's just not worth it. Simple lossless codec will be <2:1 for straight out of camera footage. Complex one will be at best 3:1, but slow to decode.
In the same time many movies were made using intermediate codecs as they simply are good enough even for big screen. Depending on the job profile you may have uncompressed master (definitely for Hollywood big titles), but there are cases where best master is actually intermediate codec. It doesn't mean much to be honest. Such a movie can still win Oscar In most cases even if there is a 16bit TIFF master most delivery masters will be made from intermediate codecs. Then they are compressed again and again and again Only few chosen deliveries will be made from main TIFF master- mainly DCP master. Netflix is picky about quality, but they still won't be able to tell if you made your IMF master from ProRes or TIFF sequence. They also later compress this crazy high quality master to 20Mbit for UHD, so whole amazing master is about pointless (you always need higher quality master compared to your end format though).

Have you actually looked what you loose with intermediate codecs? Do you know how curve for bitrate vs quality looks like?
If 50Mbit h264 can give you 99% quality then in order to get to 99.5% you may need eg 100Mbit. It's crazy non-linear curve. If codec has 5:1 mode this is already very high quality and things like 3:1 can simply replace uncompressed/lossless in real world.
For SD it's easy these days to use uncompressed or lossless. Try doing the same for UHD It's all driven by money and budgets in real world, so things like perfect scenarios basically don't happen.

Exr 16float compressed are lossless. About 2:1 compression (YMMV).

Andrew, you're not very consistent with your arguments. On one hand the space increase for lossless over intermediate is a drama for you, on the other hand the space savings over uncompressed are not worth it.

If intermediate is a perfect master for you, then just export DNx and use FFMPEG to encode to ProRes. If 1% quality loss does not matter, then 2% surly doesn't matter either for you and you get the best lossy artefacts from two codecs

OK, let's stop this for real, I already feel bad for all the people wasting their life time reading our arguments. I'm out of thread

Lossless compression for its savings adds a lot of complications. You never can guarantee compression rate- can be just 1.3x (if your frame is very noisy) or 3x. You have issues with decoding speed. EXR has few modes for lossless compression, but they are not used often as they are problematic. For all this hassle I rather had very fast and well written codec which offers 3:1 fixed compression. Then at least you know ratio, you can plan storage and bandwidth requirements, etc. Lossless compression is basically ZIP/RAR. If you want to archive you can use uncompressed and then RAR it.

Exr 16float compressed are lossless. About 2:1 compression (YMMV).

I assume Disney wasn't happy with this and introduced lossy compression, to get more savings (again- something where you can say it's 3:1 or 4:1).

Maybe a picture will help.

Here's how I envision this working. And I should mention this is basically the same approach I wish to take for all my media, including digitized 8mm/ Super 8 film, VHS, and modern video formats like AVCHD/h264, and notable cell-phone videos.

If it's about DV then capture just as DV (it's basically copying data from tape to machine). DV should survive at least as long as ffv1. You don't really need to go to uncompressed. It does nothing in your workflow, except wasting space. You can use it for analog tapes. Resolve is bad choice for your needs. It has very poor interlaced support and its deinterlacing is bad. You will have problems with video from mobile phones as well (often VBR framerate). Not sure if Resolve does import AVCHD at all.
For you something like TMPEG Mastering Works could be good choice.
AVI is also bad choice as a container (no timecode, no aspect info etc). Use MOV.

If it's about DV then capture just as DV (it's basically copying data from tape to machine). DV should survive at least as long as ffv1. You don't really need to go to uncompressed. It does nothing in your workflow, except wasting space. You can use it for analog tapes. Resolve is bad choice for your needs. It has very poor interlaced support and its deinterlacing is bad. You will have problems with video from mobile phones as well (often VBR framerate). Not sure if Resolve does import AVCHD at all.
For you something like TMPEG Mastering Works could be good choice.
AVI is also bad choice as a container (no timecode, no aspect info etc). Use MOV.

^ Basically what Andrew writes!

Exr 16float compressed are lossless. About 2:1 compression (YMMV).

I assume Disney wasn't happy with this and introduced lossy compression, to get more savings (again- something where you can say it's 3:1 or 4:1).

Yes, if you really want it and want it all, 32 float uncompressed is the way to go...

For a all baked in p3 signal, tiff 16 bit is sufficient too.

Be it organisations such as the International Association of Sound and Audiovisual Archives or Fédération Internationale des Archives de Télévision (International Federation of Television Archives) and their members.

AMIA (the Association for Moving Image Archivists), SMPTE (the Society of Motion Picture & Television Engineers), the ASRC (Association for Recorded Sound Collections) and the Library of Congress all specify WAV files or 1/4" analog tape (and often both). There's good evidence for making copies in both formats for longterm storage. The LoC's info is at this link:

http://www.digitalpreservation.gov/formats/

The Grammy Producers & Engineers' Wing has a long list of specifications on what they recommend for long term music storage:

https://www.grammy.org/files/pages/deli ... ations.pdf

http://www.grammy.org/recording-academy ... guidelines

The Academy of Motion Pictures Arts & Sciences has two tech papers on "The Digital Dilemma" talking about how hard long-term storage is for motion pictures...

https://www.oscars.org/science-technolo ... al-dilemma
https://www.oscars.org/science-technolo ... -dilemma-2

Suffice it to say... it's a long and very complex question. I think in general, even lossless compression is dicey because if there's a data error in the storage medium (like a bad sector on the drive), it can be really hard to recover the bad data. With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.

I think in general, even lossless compression is dicey because if there's a data error in the storage medium (like a bad sector on the drive), it can be really hard to recover the bad data. With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.

Detecting corruption, limiting the scope of its possible damage and ensuring robust decoding is usually very high on the design goal list of lossless codecs. Since you mentioned WAV, WavPack and FLAC are certainly prime examples that do things right in this category.

To stick with the other example mentioned in this thread, FFV1 does per frame checksums. This gives the possibility to accurately verify which frames have been affected by possible storage corruption. Those can then recovered from parity (if any available), a good copy on different media without copying the entire file again, dropped and interpolated in the very worst case.

Detecting storage corruption is these days considered the job of lower layers, the device itself or the filesystem. However modern filesystems such as ZFS, BTRFS, or ReFS are not that widely used yet.

While it's obviously a good idea to store another copy of the file somewhere else, manually adding a bit of parity can recover a lot of simple errors like bit flips or damaged sectors.

SnapRAID is a very good way to protect an entire set of discs with more or less static data. Par2 to protect against corruption on a single drive.

AMIA, SMPTE, the ASRC and the Library of Congress all specify WAV files or 1/4" analog tape (and often both). There's good evidence for making copies in both formats for longterm storage.
...
Suffice it to say... it's a long and very complex question...With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.

In other words, universally, uncompressed and lossless formats are recommended for preservation and archival of audiovisual essence.

Compressed and lossy formats are only recommended as access copies.

Best practice is to archive the source code along with essence for future playback or reconstruction, hence open source formats are preferred by archival and preservation organisations and their members.

As an active member of several of the professional bodies and societies mentioned in this thread (as well as a few not mentioned, such as the AES and the EBU), regularly in direct contact with the various task groups and standards committees in each, this issue isn’t necessarily long and complex.

The heavy lifting has been completed successfully by the various members of these organisations and outlined in depth and quite clearly in the various publications available on their respective web presence, available to read online or in physical form via the web store, if you care to do so.