Restoration of historic MTS video format

I’m starting recovery and “beautification” of a nice series of old takes (Panasonic GH2, I think). Resolve Studio 18.6.6 in current use (FCPX also an option).

Material was actually stored as unencrypted “Blue Ray disk” on the SD Card. Looks similar to real ones. Access to media is not a problem. Format of the single files are no secret either, as can be seen here:

Code: Select all

General
ID                                       : 0 (0x0)
Format                                   : BDAV
Format/Info                              : Blu-ray Video
File size                                : 186 MiB
Duration                                 : 1 min 34 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 16.6 Mb/s
Maximum Overall bit rate                 : 18.0 Mb/s
Frame rate                               : 25.000 FPS
Recorded date                            : 2011-09-11 12:01:58 UTC
Writing application                      : Panasonic

Video
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : 2 Ref Frames
Format settings, CABAC                   : No
Format settings, Reference frames        : 2 frames
Format settings, GOP                     : M=3, N=12
Codec ID                                 : 27
Duration                                 : 1 min 34 s
Bit rate mode                            : Variable
Bit rate                                 : 15.8 Mb/s
Maximum bit rate                         : 16.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First
Bits/(Pixel*Frame)                       : 0.304
Stream size                              : 177 MiB (95%)

Audio
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : 129
Duration                                 : 1 min 34 s
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 2.15 MiB (1%)
Service kind                             : Complete Main
Dialog Normalization                     : -31 dB
compr                                    : -11.02 dB
dialnorm_Average                         : -31 dB
dialnorm_Minimum                         : -31 dB
dialnorm_Maximum                         : -31 dB

Text
ID                                       : 4608 (0x1200)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144
Duration                                 : 1 min 33 s


Before committing to a specific approach, maybe someone here has a practical tip or two. Simple advice, along the lines of “don’t initially waste time on…” etc. Never hurts to listen.

Content is South Korean historical “royal guard” performance in central Seoul. Lot’s of movements, also near shots, even across the lens (opposite direction in some cases) and probably rolling shutter too in 25fps interlaced and heavily compressed 1920x1080 AVC media (MTS format). As seen in MediaInfo above.

I don’t expect the “repair, beautification, restoration” etc operation to be easy, to tell the truth. There are some really messy scene details in some takes, like several level of ’spear handlers' moving fast paced across the lens in opposite directions is not the easiest to correct in post.

Sound is surprisingly good stereo. That can be handled. Microphone far better then, than was currently expected.

Media has personal value, and no time limit is involved in restoration, so any of my machines can be dedicated for days on end (the Apple Silicon M1 mini for instance, even if ProRES is not supported on chip). Naturally AFTER verifying the approach on a few “snippets”.

Output format still to be determined. I’m not planning on saving any storage space in converted material; aiming at 1440(p) or even 2180(p), if the latter shows any visible promise in final render. Time will tell.

In some months I plan to revisit South Korea during Chuseok - one day later in the year, than in 2011 - and do a few repeat takes of personal interest, if possible. Bookings completed.

Regards

Edit: I consulted my notes and I was wrong.

All software below is only for windows:
The best way I have found is to download virtualdub, use download and use MSU Deinterlacer 2.1 with Bottom First First unticked and Double Frame Rate ticked.
Then compress with cineform or similar
Then go into Davinci with Super Scale Enhanced 2x



---- OLD INFORMATION

I did spend a bunch of time deinterlacing DVD files and the best quality I found was with ffmpeg and the nnedi filter (https://ffmpeg.org/ffmpeg-filters.html#nnedi ).

Using that filter requires you to get comfortable with the command line

ffmpeg -i INPUT-FILE -vf "nnedi=nnedi3_weights.bin:deint=all:field=a:deint=all:nsize=s48x6:nns=n256:qual=slow:pscrn=none" -c:v cfhd -quality 0 OUTPUT.MOV

Requires the .bin file to be in the directory where you run ffmpeg from.

bin file: https://github.com/dubhater/vapoursynth ... eights.bin

Quality is from 0 to 12 where 0 is the least compressed.

After the deinterlacing, I have found (as stated in viewtopic.php?f=21&t=201734#p1048010 ) Super Resolution to be a better scaling than anything else:
Download virtualdub2 ( https://sourceforge.net/projects/vdfilt ... p/download ), then download https://www.infognition.com/super_resolution_vdf/ and put it in the x64 folder in Virtualdub, then add it as a Filter, add either Cineform again or another compression.
Then you can put that resulting file into Davinci and do final touchups.

I will extend the same offer to you as I did to riptorn/bob dowd: Send me the file and I'll convert the first 10 seconds-ish so you can see if it's worth your effort.

I have not had much success deinterlacing in Davinci, so I'm ignoring that path.

Hey

I'm no more into these kind of jobs but I certainly can understand the purpose and the meaning...

Back to at least 10 years before, I used to use JDS Deinterlacer with great success each time, but I suspect it may not be ported to Silicon... (see screenshots). Searching the web would probably show anthic threads and DL option with it.

It was a solid way to deal with frames and when dealing with any mpeg2 (.vob, .m2v, .m2ts, .mts...), it was a regular stop for me. One thing I even sometimes tried in the end is to use both fields for different reasons.

Now, I think I'd try Resolve's internal basic solution or any "deinterlace" option in whatever ffmpeg-based utility I'd try and make ProRes afterwards; and probably plan a superscale thing or the same via Topaz AI

And here at least the interesting panel

I don't find the deinterlacing in DR that bad if set to 'neural'. But Topaz Video AI is better.

I’m starting recovery and “beautification” of a nice series of old takes (Panasonic GH2, I think). Resolve Studio 18.6.6 in current use (FCPX also an option). Material was actually stored as unencrypted “Blue Ray disk” on the SD Card. Looks similar to real ones. Access to media is not a problem. Format of the single files are no secret either, as can be seen here:

MTS2 is highly-compressed and awful to work with. When I have to deal with MTS, I use a 3rd party program line Wondershare Converter to change it to something I can actually work with, like H.264 or something, which can be used in Resolve. But it's still not ideal for post.

I agree with Uli above that Topaz Video AI actually does a pretty good job of deinterlacing and uprezing, but there's only so much it can do with compressed interlaced material like this. I'm not saying it's hopeless, but I would say to curb your expectations.

I can use .mts just as it is in DR. But which codec is yours exactly?
Rewrapping into .MOV might help, to avoid another round of compression.
And then, I'd go for deinterlacing and horizontal stretching only. I don't see much sense in trying to upscale to UHD at the expense of huge rendering times. Most modern TVs do a pretty good job upscaling from progressive HD, it would be hard to see much of a difference at normal viewing distance if the source material is already of limited quality.

Thank you all for the suggestions, tips and ideas.

Resolve Studio solved the basic deinterlace process for my use, no problem there.

I was more interested in the “organizational” approach for long term use (at least a few years) and storage. How to avoid unnecessary waste of time. And unplanned side effects, when used across several projects over time.

I decided to split my practical approach into two stages:

Adjustment stage

Media is de-interlaced, uprez’d, basic color adjustment and “beautification” (typically desaturating Panasonic colors, basic color balance etc.) for delivery into standard 2160p 25 fps BT.709 ProRES 422. Each original take made “usable” (trimming only start/end jitters, and obviously unusable parts etc “split off”). Media “looks” roughly adjusted to same impression. Nothing fancy, just attempting “consistent base quality”.

Sound left untreated, except normalized to EBU R128, straight conversion into 32-bit float. My standard.

Seen as a one-time “time investment”, never to be repeated in any actual project, using any of the rendered base media.

Render stage

All “adjusted” takes will be rendered fully, each as an individual 2160p ProRES 422 file. Overkill for many use cases, but now media is stable in a “unified” and “standard” portable format for most any future use (targeted at UHD 25 fps BT.709 meaning no color space conversion either).

Maybe not everything, but I have a couple of collections containing old material from all over the world worth a “clean-up”.

Combining a number of takes into one project to be rendered into converted single clip is not that complicated; also delivers a good template for later partial or complete conversions. Instead of manually doing the basic steps for each future project, where one or several clips may be used in some form or other.

Correct me, if there is something I have overlooked, but that way I have a stable “original media base” for any use in most any editor now and in the near future. Simplifies reuse in the long run.

If anything opening up (significantly) better results in the future, the project can be copied, and the new copy providing the basis for a new approach. Without affecting any existing “originals” in “unremembered” use somewhere. No old projects unexpectedly getting “quashed” or rendered “not reproducible” by accident.

I used the following settings (Resolve Studio 18.6.6). It may not be absolutely perfect, but delivers good results right off the bat (especially considering the horrible camera material) without extreme investment in “fiddling time” per individual take:

The sliders shown in default position. Worked in my case.

Resolve Studio Retime and Scaling.png (64.79 KiB) Viewed 1063 times

It may be slow, but it’s actually a good compromise for my use. Even without tweaking the sliders (that’s for a later time, when initial testing has completed).

Time is expensive, gigabytes very cheap these days.

A single 290 Mbyte file (2min 19s) becomes 7.59 GByte (uprez’ed from a horrible 1080i to an - ahem - usable 2160p). Render times roughly ten minutes - give or take - per minute (on my M1 Pro MacBook 14 Pro), but only has to be done once, for each conversion project using this approach as a template.

I can always decide to go lower to output 422LT (one click away) for a final conversion of this 16 sequences totaling 1.2 GByte original material. Haven’t tested 1440p or even 1080p results either. Yet. Have to start somewhere.

Screen dump. Lens not helping. Long end of Panasonic 14-140mm IS zoom, handheld.

Lens not helping.jpg (969.84 KiB) Viewed 1063 times

Final render (original far, far worse): Screen dump. Lens not helping. Long end of Panasonic 14-140mm IS zoom, handheld.

I Have left stabilization, sharpening, denoising, grading etc. to post in the final project. This is an attempt to produce a general “raw material conversion” template. From “siiigh”, to “ahaaa”.

Maybe a hundred dollar plus tool can do slightly better, using slightly less time, if not more; for my use, this was OK. It was USABLE. Topaz Video AI a possible alternative. Anyone knowing, where I can order buckets of patience delivered to my door, on a short time subscription basis?

As one poster alluded, the original media really was plain horrible (quality not content).

Immediate use cases

The rendered base material will initially be used as basis for two, separate projects (remaining isolated henceforth); one being an ePub project; think pdf on steroids - including text, interaction, animation, effects, audio, stills and video combined into one standalone and easily transferable “standardized” (W3C conformant) and typically read-only file. Perfect for critical off-line use.

Template use

I have loads of similar material, from Egypt etc etc etc and especially a couple of video sessions summer 2009 of the Siracusa (Sicily) water polo team in the waters separating Isola di Ortiga from Sicily are immensely precious to me (GH1 in this case). If Thai boxing seems brutal, Calcio Storico in Florence downright life threatening, these training sessions had periods of even more “dirty tricks”. Some scenes pure Tom & Jerry quality; certainly “lively” (PG rated) stuff. Nobody drowned, but… whew!

Regards

If you are dealing with already high quality footage or resotred one with not much noise or compression artifacts, SuperScale feature in Resolve Studio is really good at preserving the original when up-scaling, but not good for restaurant since it upscale and magnifies all the bad elements too. This is one place where Topaz Video AI is good, because it was trained to re-imagine compressed and bad quality footage, but its often also unpredictable so you have to watch out for changing original elements into some interpolation, and also it can be buggy, plus Topaz is still ignoring color management, so Topaz tends to average colors based on trained material. Sometimes its what you want and sometimes its not, but you can't control it. No color management.

There are communities like forums for people who try to do similar works, like https://forum.videohelp.com/ that might have good tools, there is also Resolve Studio section of plug ins that ship with it, called revival category. check that out if you haven't. Also there is Filmworkz DVO OFX Performance Pack v1.5 pack that was just released for fusion, and contains some restoration tools Hollywood used to use.

Fusion has a tone of cool things you can do for repair and restoration, but it requires familiarity with it and its best used to supplement things you do elsewhere, or combine multiple passes or repair missing frames etc. There are optical flow nodes for that.

Good luck with your project. Sounds like an interesting challenge.

IMHO, Topaz is at its best when the footage is worst. HD footage from a decent camera (and I'd definitely count the GH2 in) will not profit as much other than from deinterlacing.

IMHO, Topaz is at its best when the footage is worst. HD footage from a decent camera (and I'd definitely count the GH2 in) will not profit as much other than from deinterlacing.

That's my experience as well. And even when it comes to de-interlacing, there are some good options out there that either do as good of a job or even better than Topaz. Where Topaz is best is restoring by inventing lost details and smoothing gout the rugged edges of objects in footage.

If you want the best options then I would do the following:

Container
MTS is a container not a CODEC. If you prefer a different one remux it to mp4 or mov to avoid any reencoding.

Deinterlacing
Still, the best deinterlacing is QTGMC. Deinterlace to 50 fps as that is the correct temporal resolution of your source. You can use it in Vapoursynth, it is also available in HandBrake with a nice visual interface.

Denoising
For denoising, I prefer to use block-matching technology (BM3D). Here is an excellent implementation in CUDA: https://github.com/WolframRhodium/VapourSynth-BM3DCUDA

It has to be used in Vapoursynth, the downside is that it has a very high computational cost.

At this time neural solutions are cute but introduce all kinds of unwanted artifacts. Neural approaches are most effective at the detail level. For instance, using a neural-based dynamic chroma upsampling (4:2:0 -> 4:4:4) from motion instead of linear interpolation. Unfortunately, nothing is consumer-ready at this time.

Thank you all for the feedback and tip.

Alas, a few approaches are too complicated for my use. They can be used for one or a few files, but… it’s no reflection on the individual tools abilities, more a decision based on practical issues. For my use. YMMV.

In short: If “perfect” is not practical, then practicalities are not “perfect”.

It is not really a usable approach, when you have to convert batches of files, recorded in different light over several years, in several countries/continents and over many days (including a trek through Egypt, around Australia, through South America - Argentina, Uruguay, Paraguay, Brazil, mostly bus or ferry, only Iguazu to Rio by plane) plus… plus.

Just below 1400 files. Istanbul alone had 93 video takes of highly varying lengths. I’m NOT going to count Australia, South America, Egypt etc, where numbers are brutal - even on single days. The 16 takes from Seoul, used for initial tests, were from one, single event. As mentioned before: I had to start somewhere.

Key data are:

14-16 (max 18) megabit interlaced 1080i 25fps (each two fields) format
Total space occupied around 55.65 GigaByte

At ~2 MegaByte/sec, we’re talking roughly 27825 seconds or around 7 hours and 43 minutes of actual footage - give or take. With around 10 min render time per 1 minute original runtime, that’s roughly 77+ hours of render time. Give or take. Now we’re talking real work. Render only. In addition to initial inspection (some deleted) and “cut and brush-up” on each of the roughly 1400 takes. I'll not linger on each take, as a rule - a few especially valued takes excluded, but... I think you see my problem.

It will not be completed this month or the near future.

A nightmare to handle in any consistent way in the scenario some describe. Especially, if I need to revisit the task at a later time - let’s say in a year, if a new “wonder do-it-all setting” emerges. Even if I just wan’t to test the effect and results on a few, let’s 10 to 20, takes. Pure nightmare.

I have decided on placing all handling inside Resolve Studio Projects, covering related groups of files. Each project adjusted to the rules described earlier in the thread, and rendered to a standard ProRES format. EACH file in the project rendered individually (into a target folder for the project collection).

That way, I can decide, to do a complete re-render, if I deem it necessary, or a new option becomes available. Basic adjustments remaining the same, only requiring - let’s say - activating “new super duper AI based de-interlacing” option on the clips, then kicking the render along. However long it takes.

This can easily be done on interlaced media for my 12 travels (some multi country, nearly all multi cities, regions or places) over 2.5 years on four continents.

When organized into “easy handled” project collections, there’s light at the end of the tunnel, and to me, Resolve Studio is the best solution for my use case. It’s not perfect, but it gets the job done. Whatever elements of the original files - video, audio, converting, scaling, cutting, adjusting and normalizing - I want to “massage” - generally or on individual files - before rendering to any of a multitude of standard formats for general storage - with various use cases in other projects relying on rendered “standard” content (my standards).

Even if I want to revisit a single or few files in a collection for fine tuning, it's easily done any time.

It’s probably my best bet for a relatively future proof and project based approach, currently Apple Silicon, but who can exclude Microsoft Arm type gear in the future? Resolve Studio probably also (becoming) usable on that platform.

My personal approach. YMMV.

Regards

Why is time valued so high?

Let’s say, you are 40. You have a decent chance of living to become 80, before you “snuff it”! Unless some existing or up and coming madman interferes with your prospects.

Now, if I live to the same age, your remaining “runtime” will be five times longer, than mine!

I can’t buy time, but I can buy and use tools, that help me save time.

If you render 4:2:0 interlaced footage to ProRes 4:2:2 as a reference source you will have potential problems later on because of the chroma interpolation of not yet deinterlaced footage.

For a reference source, I would use a CODEC that either supports 4:2:0 or 4:4:4. If you still want to use ProRes you could use ProRes 4:4:4.

If you render 4:2:0 interlaced footage to ProRes 4:2:2 as a reference source you will have potential problems later on because of the chroma interpolation of not yet deinterlaced footage.

For a reference source, I would use a CODEC that either supports 4:2:0 or 4:4:4. If you still want to use ProRes you could use ProRes 4:4:4.

Thank you for the tip.

I’ll use some time testing my approach, before finalising render details.

ButnIndo not think, the problem is significant in my use case, since incoming interlaced video is converted to upscaled, progressive frame rate, rendered into ProRES. That should allow 4:2:2 use. Correct me, if I’m wrong.

Regards

If you render 4:2:0 interlaced footage to ProRes 4:2:2 as a reference source you will have potential problems later on because of the chroma interpolation of not yet deinterlaced footage.

For a reference source, I would use a CODEC that either supports 4:2:0 or 4:4:4. If you still want to use ProRes you could use ProRes 4:4:4.

Thank you for the tip.

I’ll use some time testing my approach, before finalising render details.

ButnIndo not think, the problem is significant in my use case, since incoming interlaced video is converted to upscaled, progressive frame rate, rendered into ProRES. That should allow 4:2:2 use. Correct me, if I’m wrong.

Regards

Yes, after you deinterlace(obviously before upscaling) you can use 4:2:2 or any other subsampling format.

After a bunch of “hairy” tests, I concentrated on - mainly - comparing superscale and non-superscale use.

I decided to leave out superscaling on cut basis and do the interlace settings project wide - all clips had to be de-interlaced anyway, so… easier and quicker that way.

Default clip settings used

Skærmbillede 2024-05-30 kl. 10.09.34.png (76.1 KiB) Viewed 644 times

Project settings used

Skærmbillede 2024-05-30 kl. 10.35.38.png (130.39 KiB) Viewed 644 times

It ended up as an interlace (25fps 1080i) to progressive conversion - with start/end trim and bad parts cut out. Each clip rendered individually to a project folder. No other adjustements planned (but optionally possible). Render into Rec.709 4:2:2 colorspace “autoscaled” (NO Superscale) into 2160p 25 fps ProRES 422. All audio normalized to EBU R128 converted straight to IEEE 32-bit float.

Good, solid material for inclusion into my standard project settings. In most cases only requiring colorspace trimmings, grading, maybe some sharpening and stabilization, where appropriate. The usual “cosmetics” for any take included into a project. Nothing dramatic or unusual.

Call the approach “division of labor”.

The conversion stage delivers standard stable audio and progressive video in ProRES 422 format.

The editing stage does not need to handle interlaced material, scaling or normalization etc; only the standard editing tasks always involved in post.

Example render times

Code: Select all

Runtime                     00m31s      31 sec    -
Deinterlace+trim            01m03s      63 sec    x2.03
Deinter+trim+Suprscal-min   25m18s      1518sec   x49.0
                    - max   28m01s      1681sec   x54.2

Note, that these times are only representative, for a specific case, where all tested variations of settings have been tested (one representative cut showed). De-interlace and trim runs are extremely consistent in render times, typically equal for ProRES 422LT and ProRES 422, which indicates a processing bound operation, not limited by disk write in any way. Render was to external Thunderbolt 4 SSD drive (capable of sustained 2-2.1 Gigabyte/second).

With low values for sharpening, Superscaling showed no benefit. A simple composite with the setting difference between two tracks - Superscaling on or off - resulted in uniform black (even viewed at 200 or more percent).

With higher values, there were a pronounced but odd artifact tendency; faces and yellow colors had vertical streaks and red colors horisontal streaks - more pronounced with higher sharpening values. The vertical “streaks” could not be explained as being interlace dependent. As always: YMMV and highly content dependent.

Render times now consistently in the region of a bit over two times original clip runtime. Superscaling results in an additional factor 24-27 on top of that - effectively leading to roughly a factor 50 render time compared to original runtime.

That was not acceptable. Out it went.

Project approach

The project based approach makes actual conversion (or later revised conversion) a simple procedure. A render time factor of a bit over double the realtime play factor, makes the approach a non-brainer for large scale conversion (nearly 1400 individual 1080i takes practically crying out for this - batch like - treatment)

Should a few clips need special treatment, the associated Resolve Studio conversion project only needs to be started, adjustments made, and a rerender kicked along. Simple as that. Even if it is targeted at a new version folder to avoid conflicts with existing earlier clip inclusions in previously finished projects..

Problem solved for my 1080i use cases.

As always: YMMV.

Regards

Thanks for not only testing all this, but also publishing it. I'm sure this is quite valuable information for all those in a similar situation.

For my own family HDV and Bluray files from the past I use TMPGenc MW7 to delinterlace to double frame rate ( maintain temporal motion). It is very fast and will take files directly from disc too. Helpful since I have some discs that I no longer have the source files.