Blackmagic Forum

Hi,

I manage the production facilities for a client who does film restoration. Right now the entire workflow is built around DPX. From scanning to mastering, everything is DPX. We also archive all of the scans and masters on LTO tape, and have gathered a petabyte of data.

I was wondering if we could change our workflow to something more modern, that utilises some mathematical compression to save space. DPX is entirely uncompressed, and it just seems like something we would do in the old days because CPU power was the bottleneck and DPX was universally adopted across software packages. Now the bottleneck has become storage and bandwidth. Our SAN environment is ageing and will soon need replacement, and the budgets are tight. I started looking at alternatives, such as a NAS. Unfortunately the data troughput of a NAS is not as high as that of a SAN, and doubt it cloud handle more than one 4k DPX stream. 10gbe also has not enough bandwidth to transport even a single 4k DPX stream.

So long question short: is there something else we can use other than DPX that is a lot smaller in size, but retains the original quality?

I come looking for some advice, all is welcome!

Kind regards,
Nick

EXR with compression?

Tero Ahlfors wrote:EXR with compression?

This. EXR is the best there is, different lossless compression methods (zip 1 and 16 scanline, Piz) and visually lossless DWA. Plus very flexible metadata and channel management.

Ah yes, I thought about EXR. Would there be any argument today to opt for DPX instead of compressed EXR?

Nick Verlinden wrote:Would there be any argument today to opt for DPX instead of compressed EXR?

Compatibility. You may still find DPX works in more places than EXR does, so check every part of your pipeline before switching.

Jim Simon wrote:

Nick Verlinden wrote:Would there be any argument today to opt for DPX instead of compressed EXR?

Compatibility. You may still find DPX works in more places than EXR does, so check every part of your pipeline before switching.

If some your tools not support openExr, it’s time to change. OpenExr is a vfx standard from since twenty years. I start to use it near 2001 thanks to ability to have multi layer, channel, 3d data compressed lossless to manage better workflow from software 3d to postproduction.
Just check if all your software read correctly color profile, someone also after 21 years of born do mistake on color profile embedded of openExr


Inviato dal mio iPhone utilizzando Tapatalk

Nick Verlinden wrote:I was wondering if we could change our workflow to something more modern, that utilises some mathematical compression to save space. DPX is entirely uncompressed, and it just seems like something we would do in the old days because CPU power was the bottleneck and DPX was universally adopted across software packages. Now the bottleneck has become storage and bandwidth. Our SAN environment is aging and will soon need replacement, and the budgets are tight.

I quite a bit of film restoration work here in LA, and one of our main clients does everything in ProRes 444 or 444XQ, which I would argue is "virtually lossless" (12-bit and at source resolution). You can argue that it's proprietary, but it pretty much plays on anything we throw at it and the film scans hold up very well, all the way through HDR/Dolby Vision.

The advantage of DPX is that it's open source and a good legacy format; the disadvantage is that it takes up tons of space: 2 hours of 12-bit 2K DPX is 2.25TB, while 2 hours of 12-bit 2K ProRes 444XQ is 461.6GB. I think all the major film restoration software (MTI, Diamond, Nucoda, Resolve, etc.) work fine with ProRes in and out.

OpenEXR is also open source. Depending on workflow working with image sequences can be essential or not. When it is not, any modern intra-only codec works just as good prores, dnx, cineform etc.

Thanks Marc for the info. I appreciate it.
While I do agree with you, I'm unable to sell the idea of adopting ProRes in an end to end workflow to upper management.

I'm testing some EXR workflows now with PIZ compression. In first tests I seem to gain about 42% of space.
We have a Northlight scanner, which is only able to scan DPX files, but I can convert these to EXR using ImageMagick faster than the scanner works.

I'm going to try and implement EXR once we have a replacement for our ageing Revival machines.

Thanks all for pitching in!