Archival-quality codec with lossless compression

Get answers to your questions about color grading, editing and finishing with DaVinci Resolve.
  • Author
  • Message
Offline

jdelisle

  • Posts: 31
  • Joined: Fri Mar 29, 2019 8:01 pm
  • Real Name: John Delisle

Archival-quality codec with lossless compression

PostTue Apr 02, 2019 9:24 pm

Which container format and codec are recommended for compressed lossless archival storage?

My source video is uncompressed AVI dvvideo, which I'm using ffmpeg to put in an MOV container and then edit in Resolve. My objective is to output several rendered versions, some h264/mp4 for online use, and something for long-term archival.

I'm uncomfortable using a lossy codec, and have lots of space available.

Thanks!
Offline

buckbito

  • Posts: 6
  • Joined: Tue Mar 19, 2019 6:10 pm
  • Real Name: Buck Bito

Re: Archival-quality codec with lossless compression

PostTue Apr 02, 2019 10:26 pm

Hah, I was going to suggest FFV1 in a MKV container but it looks like you already jumped on the FFV1 thread... That combo seems to be gaining some traction in the AV archival community.
Offline

Reynaud Venter

  • Posts: 2155
  • Joined: Sun May 28, 2017 9:34 am

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:53 am

Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.
Last edited by Reynaud Venter on Wed Apr 03, 2019 5:57 am, edited 1 time in total.
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:56 am

Lossless JPEG2000 is also only available for some hand picked resolutions in Resolve.
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

jdelisle

  • Posts: 31
  • Joined: Fri Mar 29, 2019 8:01 pm
  • Real Name: John Delisle

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:59 am

Reynaud Venter wrote:Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.



I'm disappointed that the best option, an open-source option, is not available directly within Resolve. I'm looking at a two-step process now, and will need considerably more disk space. :/
Offline

Reynaud Venter

  • Posts: 2155
  • Joined: Sun May 28, 2017 9:34 am

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 8:13 am

Post an addition to the DAVINCI 16 | Big feature request list:
viewtopic.php?f=21&t=71757

Blackmagic often implement user requests.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 9:16 am

jdelisle wrote:
Reynaud Venter wrote:Lossless JPEG2000 in MXF.

Unfortunately, Resolve doesn’t currently support the two most common preservation formats:
AS-07 baseband shim
FFV1 in Matroska.



I'm disappointed that the best option, an open-source option, is not available directly within Resolve. I'm looking at a two-step process now, and will need considerably more disk space. :/


No idea why you need lossless archive? In most cases it simply not needed.
What is your source format? Where does it come from? Camera recording, game, CGI etc?
Intermediate archives are already very good quality and lossless one is more for a medical etc. usage.
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 9:34 am

You could also reverse that and ask why someone should go lossy if they have no issue at all handling lossless.

This is universal. You could also say that FLAC, ALAC and WavPack are pointless and everyone should just be happy with MP3, MPC, AAC, Vorbis, Opus, etc.

Some just don't want to unnecessarily lose quality if there is no point in it. Especially if the output is only used to transcode to one more lossy format.
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

Reynaud Venter

  • Posts: 2155
  • Joined: Sun May 28, 2017 9:34 am

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 9:50 am

Andrew Kolakowski wrote:No idea why you need lossless archive?
Audiovisual archiving organisations specify lossless formats.

Be it organisations such as the International Association of Sound and Audiovisual Archives or Fédération Internationale des Archives de Télévision (International Federation of Television Archives) and their members.

Requirements adhered to by all standards compliant audiovisual archives and archival/preservation services.
Offline

jdelisle

  • Posts: 31
  • Joined: Fri Mar 29, 2019 8:01 pm
  • Real Name: John Delisle

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 4:14 pm

Andrew Kolakowski wrote:No idea why you need lossless archive? In most cases it simply not needed.
What is your source format? Where does it come from? Camera recording, game, CGI etc?
Intermediate archives are already very good quality and lossless one is more for a medical etc. usage.


My source is not the best. I'm working from 4 source formats - MiniDV, VHS, 8mm film, and Super 8 film. The majority is on MiniDV, and is transferred to my PC via Firewire as uncompressed AVI. The VHS is digitized to uncompressed AVI, and the Super 8 and 8mm are digitized using a digital camera AVCHD h264 at 1080p30.

Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:06 pm

If you would deal with HD or UHD then your would quickly change your mind :)
Your 60TB would be not much at all.
SD is easy these days.
You may as well export 10bit uncompressed YUV out of Resolve. What is the average ffv1 gain in those sources, 3:1?
Problem with ffv1 is that it's basically unsupported by pro apps. In case of restore you will need about every time convert ffv1 with ffmpeg to something usable. V210 (10bit uncompressed YUV) is universal. Works in about every tool. You can RAR it if you want to preserve space (you should gain about the same as with ffv1).
Offline
User avatar

Cary Knoop

  • Posts: 1397
  • Joined: Sun Mar 12, 2017 6:35 pm
  • Location: Newark, CA USA

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:15 pm

jdelisle wrote:Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:20 pm

peterjackson wrote:You could also reverse that and ask why someone should go lossy if they have no issue at all handling lossless.

This is universal. You could also say that FLAC, ALAC and WavPack are pointless and everyone should just be happy with MP3, MPC, AAC, Vorbis, Opus, etc.

Some just don't want to unnecessarily lose quality if there is no point in it. Especially if the output is only used to transcode to one more lossy format.


When you digitise something (specially analog) from the tape then going to something like ProRes or uncompressed gives basically same end quality. You will never see a real difference and then your digital file becomes main master (and at this point loss of eg. 1% in quality is not important at all). For SD you can easily do lossless, but for HD or UHD this is not so easy, specially if you have 1000s of hours to backup.
For mentioned MiniDV I would simply capture it as DV and keep original DV files (although I hate DV format).
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:21 pm

Cary Knoop wrote:
jdelisle wrote:Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.


For editing, cleaning up, fixing. If you have time it's good thing to do, rather than just dumping crap into your archive. Resolve for me is very bad choice for this job, but you can still use it.
Offline

MishaEngel

  • Posts: 1432
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:33 pm

Try VirtualDub2 http://virtualdub2.com/ and pick the codec you like.
Offline
User avatar

Cary Knoop

  • Posts: 1397
  • Joined: Sun Mar 12, 2017 6:35 pm
  • Location: Newark, CA USA

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:41 pm

Andrew Kolakowski wrote:
Cary Knoop wrote:
jdelisle wrote:Recognizing how difficult it is to find working MiniDV players, VHS players, and 8mm projectors, my objective is to produce archival grade output from these legacy media to last another 30-50 years. While I'll always keep the original physical and transferred digital media, I'm going through a lot of effort to correct issues like deinterlacing, correcting color, and other general edits that make the content nicer to actually watch. I'd like lossless output from Resolve to keep alongside these originals. I'll transcode that for use on Youtube or whatever, but for long-term archival, I'm not seeing an advantage to a lossy compression given that I have 60+ TB of available disk, and comparable cloud backup.

Why would you need Resolve? Archiving means you capture as good as possible and save it without any tinkering at all, even if the tinkering uses a lossless codec.


For editing, cleaning up, fixing. If you have time it's good thing to do, rather than just dumping crap into your archive. Resolve for me is very bad choice for this job, but you can still use it.

I have seen a lot of "cleaning" and "fixing" that actually destroyed information. I agree with you that Resolve is not the right tool for processing low fi interlaced material. I would recommend Vapoursynth for that.
Offline

Dermot Shane

  • Posts: 2350
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:48 pm

no issues with disk space + display refered + archive = DPX to LTO
no issues with disk space + scene refered + archive = EXR to LTO
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:50 pm

LTO takes space and costs quite big money as well :)
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 5:54 pm

Can't we just allow people wanting a lossless workflow to do things their way and people wanting a lossy workflow to do things their way.

Resolve supports a gazillion lossy variants, I don't think someone will die if it gains support for one decent lossless codec.

Different metrics have different weight to different people with different use cases on different hardware in different situations.
Last edited by peterjackson on Wed Apr 03, 2019 7:12 pm, edited 1 time in total.
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

MishaEngel

  • Posts: 1432
  • Joined: Wed Aug 29, 2018 12:18 am
  • Real Name: Misha Engel

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:03 pm

When you want better watchable quality than the originals, this might be a solution https://www.redsharknews.com/production/item/6227-how-ai-is-helping-to-make-the-best-classic-tv-series-restorations for some of your source material.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:07 pm

peterjackson wrote:Why can't we just allow people wanting a lossless workflow to do things their way and people wanting a lossy workflow to do things their way.

Resolve supports a gazillion lossy variants, I don't think someone will die if it gains support for one decent lossless codec.

Different metrics have different weight to different people with different use cases on different hardware in different situations.


No one will die. No one will protest I assume, but list of the new features "needed in Resolve is about endless.
I 10000x prefer to see ProRes export than ffv1.
Maybe there is a reason for gazillion lossy formats and none of the lossless ones :D
Offline

jdelisle

  • Posts: 31
  • Joined: Fri Mar 29, 2019 8:01 pm
  • Real Name: John Delisle

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:18 pm

Andrew Kolakowski wrote:Maybe there is a reason for gazillion lossy formats and none of the lossless ones :D


I'm curious how that makes sense, it feels like I must be misunderstanding something here.

If I were a professional working on e.g. a movie, I'd want to retain as many artifacts as possible in the best possible quality. I'd keep all the original footage and everything I render through Resolve in a lossless compressed format. This way, in 10 years even if I cannot reproduce the render using Resolve, I still have the master render to use as an input to further edits/ improvements etc. in any tool I wish. Why would I want to start from a lossy version? It'd be like a photocopy of a photocopy.. each time losing quality and original data.

Forgive my lack of insight into the video industry, but assuming the final movie output is rendered by Resolve, wouldn't a lossless compressed master and a lossy compressed versions be required by the client, or for my own safe keeping?

What are pros delivering when say a 4k Bluray is getting pressed? Lossy codecs?
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:26 pm

Andrew Kolakowski wrote: No one will die. No one will protest I assume, but list of the new features "needed in Resolve is about endless.
OK, so let's both pitch for FFMPEG. That way we can both be happy ;)
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:32 pm

jdelisle wrote:
Andrew Kolakowski wrote:Maybe there is a reason for gazillion lossy formats and none of the lossless ones :D


I'm curious how that makes sense, it feels like I must be misunderstanding something here.

If I were a professional working on e.g. a movie, I'd want to retain as many artifacts as possible in the best possible quality. I'd keep all the original footage and everything I render through Resolve in a lossless compressed format. This way, in 10 years even if I cannot reproduce the render using Resolve, I still have the master render to use as an input to further edits/ improvements etc. in any tool I wish. Why would I want to start from a lossy version? It'd be like a photocopy of a photocopy.. each time losing quality and original data.

Forgive my lack of insight into the video industry, but assuming the final movie output is rendered by Resolve, wouldn't a lossless compressed master and a lossy compressed versions be required by the client, or for my own safe keeping?

What are pros delivering when say a 4k Bluray is getting pressed? Lossy codecs?


Pro movies use DPX or EXRs so this is uncompressed (from 10 to32bit float precision). No one uses lossless as it's just not worth it. Simple lossless codec will be <2:1 for straight out of camera footage. Complex one will be at best 3:1, but slow to decode.
In the same time many movies were made using intermediate codecs as they simply are good enough even for big screen. Depending on the job profile you may have uncompressed master (definitely for Hollywood big titles), but there are cases where best master is actually intermediate codec. It doesn't mean much to be honest. Such a movie can still win Oscar :) In most cases even if there is a 16bit TIFF master most delivery masters will be made from intermediate codecs. Then they are compressed again and again and again :) Only few chosen deliveries will be made from main TIFF master- mainly DCP master. Netflix is picky about quality, but they still won't be able to tell if you made your IMF master from ProRes or TIFF sequence. They also later compress this crazy high quality master to 20Mbit for UHD, so whole amazing master is about pointless (you always need higher quality master compared to your end format though).

Have you actually looked what you loose with intermediate codecs? Do you know how curve for bitrate vs quality looks like?
If 50Mbit h264 can give you 99% quality then in order to get to 99.5% you may need eg 100Mbit. It's crazy non-linear curve. If codec has 5:1 mode this is already very high quality and things like 3:1 can simply replace uncompressed/lossless in real world.
For SD it's easy these days to use uncompressed or lossless. Try doing the same for UHD :) It's all driven by money and budgets in real world, so things like perfect scenarios basically don't happen.
Last edited by Andrew Kolakowski on Wed Apr 03, 2019 7:48 pm, edited 3 times in total.
Offline
User avatar

waltervolpatto

  • Posts: 8660
  • Joined: Thu Feb 07, 2013 5:07 pm
  • Location: 1146 North Las Palmas Ave. Hollywood, California 90038 USA

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:46 pm

Exr 16float compressed are lossless. About 2:1 compression (YMMV).
SuperServer 5039AD-I
C9X299-PGF - DDR4-2400 16x4 GB
i9-7920xCPU 12c 2.90GHz Water cooled
2x 1080ti DeckLink Studio 4K (11.5.1)
W10-1903 - BMR St. 17.2.011
nvidia: 462.59 studio
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:47 pm

Andrew, you're not very consistent with your arguments. On one hand the space increase for lossless over intermediate is a drama for you, on the other hand the space savings over uncompressed are not worth it.

If intermediate is a perfect master for you, then just export DNx and use FFMPEG to encode to ProRes. If 1% quality loss does not matter, then 2% surly doesn't matter either for you and you get the best lossy artefacts from two codecs :D

OK, let's stop this for real, I already feel bad for all the people wasting their life time reading our arguments. I'm out of thread ;)
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 7:55 pm

Lossless compression for its savings adds a lot of complications. You never can guarantee compression rate- can be just 1.3x (if your frame is very noisy) or 3x. You have issues with decoding speed. EXR has few modes for lossless compression, but they are not used often as they are problematic. For all this hassle I rather had very fast and well written codec which offers 3:1 fixed compression. Then at least you know ratio, you can plan storage and bandwidth requirements, etc. Lossless compression is basically ZIP/RAR. If you want to archive you can use uncompressed and then RAR it.
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 8:02 pm

waltervolpatto wrote:Exr 16float compressed are lossless. About 2:1 compression (YMMV).


I assume Disney wasn't happy with this and introduced lossy compression, to get more savings (again- something where you can say it's 3:1 or 4:1).
Offline

jdelisle

  • Posts: 31
  • Joined: Fri Mar 29, 2019 8:01 pm
  • Real Name: John Delisle

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 8:15 pm

Maybe a picture will help.

Here's how I envision this working. And I should mention this is basically the same approach I wish to take for all my media, including digitized 8mm/ Super 8 film, VHS, and modern video formats like AVCHD/h264, and notable cell-phone videos.

Image
Offline

Andrew Kolakowski

  • Posts: 7389
  • Joined: Tue Sep 11, 2012 10:20 am
  • Location: Poland

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 8:22 pm

If it's about DV then capture just as DV (it's basically copying data from tape to machine). DV should survive at least as long as ffv1. You don't really need to go to uncompressed. It does nothing in your workflow, except wasting space. You can use it for analog tapes. Resolve is bad choice for your needs. It has very poor interlaced support and its deinterlacing is bad. You will have problems with video from mobile phones as well (often VBR framerate). Not sure if Resolve does import AVCHD at all.
For you something like TMPEG Mastering Works could be good choice.
AVI is also bad choice as a container (no timecode, no aspect info etc). Use MOV.
Offline
User avatar

Cary Knoop

  • Posts: 1397
  • Joined: Sun Mar 12, 2017 6:35 pm
  • Location: Newark, CA USA

Re: Archival-quality codec with lossless compression

PostWed Apr 03, 2019 8:30 pm

Andrew Kolakowski wrote:If it's about DV then capture just as DV (it's basically copying data from tape to machine). DV should survive at least as long as ffv1. You don't really need to go to uncompressed. It does nothing in your workflow, except wasting space. You can use it for analog tapes. Resolve is bad choice for your needs. It has very poor interlaced support and its deinterlacing is bad. You will have problems with video from mobile phones as well (often VBR framerate). Not sure if Resolve does import AVCHD at all.
For you something like TMPEG Mastering Works could be good choice.
AVI is also bad choice as a container (no timecode, no aspect info etc). Use MOV.

^ Basically what Andrew writes!
Offline
User avatar

waltervolpatto

  • Posts: 8660
  • Joined: Thu Feb 07, 2013 5:07 pm
  • Location: 1146 North Las Palmas Ave. Hollywood, California 90038 USA

Re: Archival-quality codec with lossless compression

PostThu Apr 04, 2019 12:12 am

Andrew Kolakowski wrote:
waltervolpatto wrote:Exr 16float compressed are lossless. About 2:1 compression (YMMV).


I assume Disney wasn't happy with this and introduced lossy compression, to get more savings (again- something where you can say it's 3:1 or 4:1).


Yes, if you really want it and want it all, 32 float uncompressed is the way to go...

For a all baked in p3 signal, tiff 16 bit is sufficient too.
SuperServer 5039AD-I
C9X299-PGF - DDR4-2400 16x4 GB
i9-7920xCPU 12c 2.90GHz Water cooled
2x 1080ti DeckLink Studio 4K (11.5.1)
W10-1903 - BMR St. 17.2.011
nvidia: 462.59 studio
Offline
User avatar

Marc Wielage

  • Posts: 7361
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Archival-quality codec with lossless compression

PostThu Apr 04, 2019 3:50 am

Reynaud Venter wrote:Be it organisations such as the International Association of Sound and Audiovisual Archives or Fédération Internationale des Archives de Télévision (International Federation of Television Archives) and their members.

AMIA (the Association for Moving Image Archivists), SMPTE (the Society of Motion Picture & Television Engineers), the ASRC (Association for Recorded Sound Collections) and the Library of Congress all specify WAV files or 1/4" analog tape (and often both). There's good evidence for making copies in both formats for longterm storage. The LoC's info is at this link:

http://www.digitalpreservation.gov/formats/

The Grammy Producers & Engineers' Wing has a long list of specifications on what they recommend for long term music storage:

https://www.grammy.org/files/pages/deli ... ations.pdf

http://www.grammy.org/recording-academy ... guidelines

The Academy of Motion Pictures Arts & Sciences has two tech papers on "The Digital Dilemma" talking about how hard long-term storage is for motion pictures...

https://www.oscars.org/science-technolo ... al-dilemma
https://www.oscars.org/science-technolo ... -dilemma-2

Suffice it to say... it's a long and very complex question. I think in general, even lossless compression is dicey because if there's a data error in the storage medium (like a bad sector on the drive), it can be really hard to recover the bad data. With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

peterjackson

  • Posts: 595
  • Joined: Sat Aug 18, 2018 7:12 pm
  • Real Name: Peter Jackson

Re: Archival-quality codec with lossless compression

PostThu Apr 04, 2019 4:47 am

Marc Wielage wrote:I think in general, even lossless compression is dicey because if there's a data error in the storage medium (like a bad sector on the drive), it can be really hard to recover the bad data. With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.
Detecting corruption, limiting the scope of its possible damage and ensuring robust decoding is usually very high on the design goal list of lossless codecs. Since you mentioned WAV, WavPack and FLAC are certainly prime examples that do things right in this category.

To stick with the other example mentioned in this thread, FFV1 does per frame checksums. This gives the possibility to accurately verify which frames have been affected by possible storage corruption. Those can then recovered from parity (if any available), a good copy on different media without copying the entire file again, dropped and interpolated in the very worst case.

Detecting storage corruption is these days considered the job of lower layers, the device itself or the filesystem. However modern filesystems such as ZFS, BTRFS, or ReFS are not that widely used yet.

While it's obviously a good idea to store another copy of the file somewhere else, manually adding a bit of parity can recover a lot of simple errors like bit flips or damaged sectors.

SnapRAID is a very good way to protect an entire set of discs with more or less static data. Par2 to protect against corruption on a single drive.
Last edited by peterjackson on Thu Apr 04, 2019 7:13 am, edited 8 times in total.
XPS 9750, i9-8950HK, 32GB, 2TB NVM, Intel 630, GTX 1050 Ti, 1080 Ti eGPU, Mini Monitor, BMCC4K. Versions: BIOS 1.7.0, Intel 630 24.20.100.6287, Nvidia 25.21.14.1881, Thunderbolt 17.4.77.13, Win10 Enterprise 10.0.17134.
Offline

Reynaud Venter

  • Posts: 2155
  • Joined: Sun May 28, 2017 9:34 am

Re: Archival-quality codec with lossless compression

PostThu Apr 04, 2019 5:50 am

Marc Wielage wrote:AMIA, SMPTE, the ASRC and the Library of Congress all specify WAV files or 1/4" analog tape (and often both). There's good evidence for making copies in both formats for longterm storage.
...
Suffice it to say... it's a long and very complex question...With predictable fixed-bitrate files like WAVs, it's easier for recovery software to know where the next group of data is going to be.
In other words, universally, uncompressed and lossless formats are recommended for preservation and archival of audiovisual essence.

Compressed and lossy formats are only recommended as access copies.

Best practice is to archive the source code along with essence for future playback or reconstruction, hence open source formats are preferred by archival and preservation organisations and their members.

As an active member of several of the professional bodies and societies mentioned in this thread (as well as a few not mentioned, such as the AES and the EBU), regularly in direct contact with the various task groups and standards committees in each, this issue isn’t necessarily long and complex.

The heavy lifting has been completed successfully by the various members of these organisations and outlined in depth and quite clearly in the various publications available on their respective web presence, available to read online or in physical form via the web store, if you care to do so.

Return to DaVinci Resolve

Who is online

Users browsing this forum: AndreeMarkefors, Bing [Bot], David E King, Google [Bot], newb0001, Nicolaas Cottenie, Thomas Dove and 86 guests