Like Project Sound Lift

  • Author
  • Message
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Like Project Sound Lift

PostSat Dec 02, 2023 1:23 am

Like Adobe's upcoming Project Sound Lift, DVR (and frankly every other NLE) should have the ability to separate production. tracks into their various elements, be they individual voices, background noise, cars, airplanes, etc., including separating musical instruments into stems.
Offline
User avatar

Marc Wielage

  • Posts: 10774
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Like Project Sound Lift

PostSat Dec 02, 2023 3:56 am

That's a complicated subject, and I count the following Digital Extraction software companies doing it now:

http://www.algorithmix.com/en/renovator.htm
https://audionamix.com/xtrax-stems
https://www.audiosourcere.com/demix-pro ... -software/
https://hitnmix.com/remix-software
https://www.izotope.com/en/products/rx.html
http://www.prosoniq.com/editing-products/sonicworx/
https://www.steinberg.net/spectralayers/

A good friend of mine, Chris Kissel, has an entire website devoted to the concept of digitally extracting (separating and "unmixing") previously-mixed tracks, even mono tracks:

https://www.monotostereo.info/

It's a very interesting subject, but the problem is that it doesn't always work, sometimes there are terrible artifacts (almost trainwrecks), and it often requires a lot of human intervention and judgement. But when it does work, it's pretty miraculous.

The noise reduction tools out there work on similar principles: identify the sounds we want to keep, and remove (or lower) what we don't want to hear:

https://www.accentize.com/dxrevive
https://acondigital.com/products/restoration-suite/
http://www.izotope.com/products/audio/rx/
https://absentiadx.com/
https://goyo.app/
https://www.waves.com/clarity-vx-vocal-noise-reduction

There's a lot of overlap in the area between digital noise reduction and digital "layer" separation. I don't know if it's something BMD wants to get into, mostly because it'll be a potential bloated plug-in dragging down software efficiency. My experience is that it's problematic even in Pro Tools (which only does audio).
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Re: Like Project Sound Lift

PostSun Dec 03, 2023 6:47 am

Timber wrote:Like Adobe's upcoming Project Sound Lift, DVR (and frankly every other NLE) should have the ability to separate production. tracks into their various elements, be they individual voices, background noise, cars, airplanes, etc., including separating musical instruments into stems.


I'm presuming that competitive pressure will force the issue. It seems pretty obvious that it will be a standard feature in every NLE by this time next year. It'll just be another reason we need a new computer with double the resources of our currently adequate machines.
Offline
User avatar

Marc Wielage

  • Posts: 10774
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Like Project Sound Lift

PostMon Dec 04, 2023 3:34 am

Timber wrote:I'm presuming that competitive pressure will force the issue. It seems pretty obvious that it will be a standard feature in every NLE by this time next year. It'll just be another reason we need a new computer with double the resources of our currently adequate machines.

The problem with that is, look at the number of trouble reports on people trying to use Voice Isolate, which is a relatively-simple plug-in that just removes extraneous noise from location dialogue. That's stressful on systems, so trust me, getting something that "unmixes" multiple things at one time is even more difficult.

I can run the iZotope RX 10 "Music Rebalance" plug-in in standalone mode on my Mac laptop, but it basically takes about 5 minutes to process a 3-minute song, and it gives me 4 different tracks:

Image

Trying to "train" a plug-in to identify one voice from another, or a nearby train, or car noise, or crowd cheering in the background... that would be very tricky. It's not an automatic operation and would require lots of controls along with time and effort. And it'd be a computationally-intense operation. If you try to do that with color-correction and editing and transitions (and god knows, a Fusion composition), all during a render... I think it could result in a nuclear melt down. Resolve can't do everything or be everything to everybody. My joke years ago is "hey, they could add a word processor and a spreadsheet, and you could actually run a business on it!" But there are limitations.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Re: Like Project Sound Lift

PostWed Dec 06, 2023 11:36 pm

IMHO the speed problems are relative, and fleeting. I remember waiting 5 minutes for Photoshop to process a 601 frame in Gallery Effects; something one could scrub through of play in real time now.
Offline
User avatar

Marc Wielage

  • Posts: 10774
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Like Project Sound Lift

PostWed Dec 06, 2023 11:55 pm

Timber wrote:IMHO the speed problems are relative, and fleeting. I remember waiting 5 minutes for Photoshop to process a 601 frame in Gallery Effects; something one could scrub through of play in real time now.

Tell me how fast Voice Isolate runs now on your system. Now imagine sound processing 5 or 10 times more complex.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

Jim Simon

  • Posts: 29409
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Like Project Sound Lift

PostThu Dec 07, 2023 2:15 pm

In professional production, wouldn't all of things things be recorded separately already?
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Re: Like Project Sound Lift

PostThu Dec 07, 2023 8:17 pm

Jim Simon wrote:In professional production, wouldn't all of things things be recorded separately already?


For scripted drama, sure. But think of how much material is reality or documentary these days. I do agree with Marc that the processing demands are on the ragged edge of the possible with today's computers, but besides the point that they aren't for tomorrow's computers, that's also why we have render caches now. I can easily build a Fusion Clip that will drag my computer to a crawl, but cache the result and it plays just fine.
Offline
User avatar

Marc Wielage

  • Posts: 10774
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Like Project Sound Lift

PostFri Dec 08, 2023 3:58 am

Timber wrote:For scripted drama, sure. But think of how much material is reality or documentary these days. I do agree with Marc that the processing demands are on the ragged edge of the possible with today's computers, but besides the point that they aren't for tomorrow's computers, that's also why we have render caches now. I can easily build a Fusion Clip that will drag my computer to a crawl, but cache the result and it plays just fine.

Well, even for unscripted content and documentaries, you can take selected scenes, then render those audio files out, drop them into iZotope RX (using a preset), and let them process outside of Resolve. And when you're done, bring the processed file in on an adjacent track and see if that works better than the original.

I think the idea for Dialogue Isolate is good if you just need a casual hand at cleaning up some extraneous noise, but there are situations where the noise (or music bleed or traffic sounds) are so severe, you have to break out the big guns. I had a case some years ago where I was helping out a friend with a small indie feature, and he had a scene shot in a public park that was perfect except for a garbage truck beep-beep-beeping while backing up in the background of the shot. I told him I was sure we didn't need to reshoot (or ADR) the scene, and I was able to surgically remove the beeps without touching the dialogue. At the time (2015), iZotope RX was the only affordable way to do this. Now, there's a lot of options out there, but I wouldn't expect them to run optimally as Resolve plug-ins.
marc wielage, csi • VP/color & workflow • chroma | hollywood
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Re: Like Project Sound Lift

PostFri Dec 08, 2023 8:45 pm

Marc, your example beeps brings up another audio FR of mine - spectral editing. Most DAW's these days (Audition, for example) one is able to paint/erase/blend into a spectral plot, and easily remove sounds which are easily identifiable like beeps, shoe squeaks and the like. Not in Fairlight, though.
Offline

Jim Simon

  • Posts: 29409
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Like Project Sound Lift

PostFri Dec 08, 2023 8:59 pm

Timber wrote:But think of how much material is reality or documentary these days.
Still...you Isolate the voice (if there is one) and then Foley or SFX the rest.

Isn't that how it's done?
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Timber

  • Posts: 124
  • Joined: Sat Feb 09, 2019 5:23 am
  • Real Name: Tim Sassoon

Re: Like Project Sound Lift

PostTue Dec 12, 2023 6:46 pm

Jim Simon wrote:
Timber wrote:But think of how much material is reality or documentary these days.
Still...you Isolate the voice (if there is one) and then Foley or SFX the rest.

Isn't that how it's done?


If there's the budget. The main thing AI is doing is bringing higher-end production values down the food chain. When I said "competitive pressure", here's what Adobe is _publicly_ announcing for Premier and After Effects (quoted from their PR):

Featured AI Tools:
• Text-Based Editing: Revolutionize your workflow with AI-driven text editing, making the process smoother and more dynamic than ever.
• Adobe Podcast Enhance: Elevate your audio content effortlessly, ensuring top-notch quality for your podcasts with Adobe's AI enhancements.
• Auto Reframe: Seamlessly adapt your content to different aspect ratios with the power of Auto Reframe, saving you time and effort in the editing process.
• Auto Ducking: Let AI handle audio balancing as you focus on the creative aspects, ensuring a professional sound mix effortlessly.
• Remix Tool: Explore the Remix Tool's innovative approach to audio editing, allowing for quick and easy adjustments to match your vision.
• Scene Detection: Effortlessly navigate through your footage with AI-driven scene detection, streamlining the editing process and maximizing efficiency.
• Auto-Match Color: Achieve a consistent and polished look across your project with Auto-Match Color, simplifying color grading with AI precision.
Offline
User avatar

Robert Niessner

  • Posts: 4907
  • Joined: Thu Feb 21, 2013 9:51 am
  • Location: Graz, Austria

Re: Like Project Sound Lift

PostWed Dec 13, 2023 7:21 am

Marc Wielage wrote:
Timber wrote:I'm presuming that competitive pressure will force the issue. It seems pretty obvious that it will be a standard feature in every NLE by this time next year. It'll just be another reason we need a new computer with double the resources of our currently adequate machines.

The problem with that is, look at the number of trouble reports on people trying to use Voice Isolate, which is a relatively-simple plug-in that just removes extraneous noise from location dialogue. That's stressful on systems, so trust me, getting something that "unmixes" multiple things at one time is even more difficult.

I can run the iZotope RX 10 "Music Rebalance" plug-in in standalone mode on my Mac laptop, but it basically takes about 5 minutes to process a 3-minute song, and it gives me 4 different tracks:

Image

Trying to "train" a plug-in to identify one voice from another, or a nearby train, or car noise, or crowd cheering in the background... that would be very tricky. It's not an automatic operation and would require lots of controls along with time and effort. And it'd be a computationally-intense operation. If you try to do that with color-correction and editing and transitions (and god knows, a Fusion composition), all during a render... I think it could result in a nuclear melt down. Resolve can't do everything or be everything to everybody. My joke years ago is "hey, they could add a word processor and a spreadsheet, and you could actually run a business on it!" But there are limitations.


Well Marc, Music Rebalance now runs in real-time in iZotope Ozone 11.

The main problem with iZotope is their complete inability to optimize their software code.
Several tools in RX still do not use more than two cores. They have yet to improve one of the most important tools of RX - the selection tool (I wrote a long list of suggestions of how to make the tool much more useful and powerful - well that was when RX 8 was new. They thanked me for the great ideas and never did anything with it). Their voice transcript tool is laughable bad and has never been improved.
Several plugins do hog the GPU at 100% while doing nothing but just sitting in the background.
Put a few instances of Neutron 4 into Fairlight tracks and you will grind to Resolve to a halt, although not processing anything. Last update of RX 10.4.2 crashes like never before, quite often just while moving the mouse over the gain window. It’s been 3 months without an update although support told me they are aware of this issue.
Saying "Thx for help!" is not a crime.
--------------------------------
Robert Niessner
LAUFBILDkommission
Graz / Austria
--------------------------------
Blackmagic Camera Blog (German):
http://laufbildkommission.wordpress.com

Read the blog in English via Google Translate:
http://tinyurl.com/pjf6a3m

Return to DaVinci Resolve Feature Requests

Who is online

Users browsing this forum: No registered users and 6 guests