Blackmagic Forum

Full transparency, this is a repost from my Reddit post.

I bought Studio 19 to get away from CapCut, and the transcription feature in resolve was one big selling points for this.

However, I am discovering that there are some weird errors even in version 20.

When speaking, I sometimes end up stuttering or saying some things over before getting it just right.

So for example, if I say "The results are... the results are obviousl... The results are obviously false." The transcription will just say "The results are obviously false." And when I'm repeating myself, the transcription will stop on the word "are" and not move to the next one until I say it correctly.

It's odd, because it's as if the transcription's AI is correcting what I'm saying in text, but those corrections are not being carried over to the video edits.

If I highlight that sentence and place it in the timeline, it will take the entire phrase, mistakes and all, and I'll end up having to edit them out by hand.

As a test, I imported the video into a few other platforms that have this feature and they don't correct the words, but write out everything that I'm saying. That allows me to properly edit out the mistakes.

It's as if Resolve is transcribing what SHOULD be said, and not what's ACTUALLY being said.

Has anyone else experienced this in Resolve 20 (or 19 for that matter)?

Are there some settings I can tweak to get rid of this?

I even exported the script and tried using the Inteliscript feature to see if Resolve would find the proper edits and cut the video properly... it did not. It basically did the same thing as before and left in all of the mistakes.

Any advice is appreciated. I'd hate to go back to another software for this, but if there's no way to avoid this, then I suppose I'll have to.

THanks!

Step 1 is to post the sound file.

But I just tested, with https://drive.google.com/file/d/121NB-A ... drive_link and Davinci transcribed correctly.

This is a test of the Da Vinci transcription engine.

(...)

The results are... The result are obviously... The result are obviously false.

David, I'm not native English speaker, i know that "any" software will not get my ItalEnglish correctly: do you have any particular inflection?

No, I have a general midwestern accent (the kind you hear on news stations like CNN)

Apologies for not posting a specific example.

Mads, I imported your audio and it did transcribe correctly.

I'll post my example here.
https://drive.google.com/file/d/1jwV9NL ... sp=sharing

Adversely, here is a screenshot of dialogue from the same unedited transcription in another application.
https://drive.google.com/file/d/1NqDzsV ... sp=sharing

This is only one example of this happening. It happens all over the project.

I'm using Resolve 20 build 49 which as of this writing, is the latest update available.

Hopefully tis gives a bit more detail into what I mean.

You can try the audio file for yourself.
https://drive.google.com/file/d/1IOEO6L ... sp=sharing

Thanks for your help.

I believe I've seen that before, Resolve basically understands I am messing up a bit and only shows the final sentence. I thought it was a good thing? I am on 19 though.

Tony359 wrote:I believe I've seen that before, Resolve basically understands I am messing up a bit and only shows the final sentence. I thought it was a good thing? I am on 19 though.

I thought the same thing!... But then I pulled the selected text into the timeline, and it kept all the mistakes.

Check out the video I linked above. I still end up having to edit out the errors.

Maybe I'm missing something.

I don't think subtitles update in real time - I always generate the subtitles at the end, after everything has been edited.

Youre right. The subtitles don't do this.

I'm referring to pulling text in from the transcription window.



This text does not properly represent what I said in the recording. So when I highlight this text from the transcription window and import the resulting video edit into the timeline, all of the errors are still there.

If this were a single occurrence or even happened just a few times, that'd be okay. But it happens many times throughout the project. I've linked to an example in the post above.

I've tested a few other videos and honestly it's kind of hit and miss. But it seems the AI model it's using is guessing what's supposed to be said and not hearing what's actually being said.

I watched the video again.

I think what you are expecting would be to Resolve to precisely match your script, errors included so you can then edit the text rather than the clip?

Resolve instead "makes your subtitles better" and you'd expect it to paste an edited version of your clip based on the polished subtitle track.

Do I understand that right?

I never considered editing based on subtitles to be honest. Unless I am mistaken though, I don't think Resolve can do that. You might be looking for a feature which is not there. Happy to be proven wrong.

Resolve can not create transcription or subtitles from this video but if you cut the ambient noise at the beginning it will do fine. I've seen this countless times, the noise breaks the AI.

https://www.swisstransfer.com/d/0d64b8b ... 50294145d4

but there is no noise in the example posted by the OP?

[/quote]

Tony359 wrote:I watched the video again.

I think what you are expecting would be to Resolve to precisely match your script, errors included so you can then edit the text rather than the clip?

Resolve instead "makes your subtitles better" and you'd expect it to paste an edited version of your clip based on the polished subtitle track.

Do I understand that right?

I never considered editing based on subtitles to be honest. Unless I am mistaken though, I don't think Resolve can do that. You might be looking for a feature which is not there. Happy to be proven wrong.

Thanks for taking a look at the example. Yes, that is what I'm expecting Resolve to do.

Resolve does actually have this as a feature. (You see me do this in the video, although it has the errors in it.)

This feature is called Text-based editing. It's actually been in Resolve since 18.5


There are other videos that go more indepth, but this is how the feature is used.

CougerJoe wrote:Resolve can not create transcription or subtitles from this video but if you cut the ambient noise at the beginning it will do fine. I've seen this countless times, the noise breaks the AI.

https://www.swisstransfer.com/d/0d64b8b ... 50294145d4

Thanks for your reply, but my video has no noise in it. Also, this is a song and not someone talking naturally. So the use-case is a bit different. Take a look at the example I posted and you'll see what I mean.

Thanks for the video link

I think I watched that video when the feature was released but then I forgot to test it! It might be useful for me too.

I am not familiar with the "transcript" tool - I only generate subtitles.
With subtitles, if Resolve "makes a sentence better", that is a good thing. In the transcript, it might not as you are experiencing.

I am not really sure whether that is intentional or not I'm afraid. Sorry again, I thought you were talking about subtitles.

I'll go test it now

yeah check it out! It's still pretty good. It's just a niusance to go back and fix things I know it should be able to handle based on many other apps with the same feature handling it just fine.

I gave it a go - it doesn't cut very precisely but it's an interesting tool.

In my case I feel it was pretty accurate - I am not scripted and I have an accent so possibly the worst case scenario! I'm on 19 though.

That sounds about right actually.

I rarely have scripted videos and I say things in a silly way sometimes and it's semi-accurate. The cuts are a little rough sometimes. Granted, still a lot easier than editing by hand... but still a little ways to go before it reaches some of the other variations of this in other programs.

I'll definitely give that a go - thanks for reminding that feature to me!

ViRiX Dreamcore wrote:

CougerJoe wrote:Resolve can not create transcription or subtitles from this video but if you cut the ambient noise at the beginning it will do fine. I've seen this countless times, the noise breaks the AI.

https://www.swisstransfer.com/d/0d64b8b ... 50294145d4

Thanks for your reply, but my video has no noise in it. Also, this is a song and not someone talking naturally. So the use-case is a bit different. Take a look at the example I posted and you'll see what I mean.

I replied to wrong thread, there was someone talking about how they make lyric videos (assumed for music videos) and how large parts of lyrics are not transcribed when to their ear the singers audio is very easy to understand. Non human speech noise(not where the singers audio is) seems to be able to break the AI for short, medium term or completely. If they fix that bug maybe it fixes your problem for reasons not understood, but having posted in wrong thread I haven't read/ downloaded your example, but I shall and I"ll see if I have anything extra to offer other than what's already been well said.

I had a look at your opening comment, what Resolve does in condensing your speech whisper faster also does.


It has many options such as repetition and repeat which could be controlling what you're seeing in Resolve but no idea if Resolve is using one of the whisper variants. I do prefer how they condense, I don't want 100% literal transcribing but then you have problems such as your experiencing with text based editing, you need more control.

Oh! lol that explains it.

Yeah, that makes sense. What the person could do is use a vocal isolation AI to seperate out the vocals (or if they have the original stems, just place those in by themselves) and it should work fine.

EDIT: Aaahh so it's a setting, but it seems we don't have access to this in the base Resolve transcription feature. I've seen many talking about using Whisper, but never figured out how to set it up. If there was a UI plugin for it, that'd be amazing. I'm not very good at scripting, but I'll look into this!

I wish they would give us a bit of settings for doing this if they are indeed using whisper.

Thanks for looking into this! That sheds a bit more light on what could be happening. And yes, you're correct. Since I"m editing the dialogue through text, I'll need it to be pretty accurate to what I'm saying.

I've tested the feature myself - it's AMAZING! But it does what you say.

It condenses sentences when I repeat myself and appends the repetitions as well, making it less amazing.
Yes, Transcribe must be accurate or it's pointless. I hope BMD improves on that because this is amazing.

@ViRiX Dreamcore

I've added a feature request here: viewtopic.php?f=33&t=222949

Tony359 wrote:I've tested the feature myself - it's AMAZING! But it does what you say.

It condenses sentences when I repeat myself and appends the repetitions as well, making it less amazing.
Yes, Transcribe must be accurate or it's pointless. I hope BMD improves on that because this is amazing.

It really is! And thanks! I'll support it. Hopefully they update and improve it.

CougerJoe wrote:

ViRiX Dreamcore wrote:

CougerJoe wrote:Resolve can not create transcription or subtitles from this video but if you cut the ambient noise at the beginning it will do fine. I've seen this countless times, the noise breaks the AI.

https://www.swisstransfer.com/d/0d64b8b ... 50294145d4

Thanks for your reply, but my video has no noise in it. Also, this is a song and not someone talking naturally. So the use-case is a bit different. Take a look at the example I posted and you'll see what I mean.

I replied to wrong thread, there was someone talking about how they make lyric videos (assumed for music videos) and how large parts of lyrics are not transcribed when to their ear the singers audio is very easy to understand. Non human speech noise(not where the singers audio is) seems to be able to break the AI for short, medium term or completely. If they fix that bug maybe it fixes your problem for reasons not understood, but having posted in wrong thread I haven't read/ downloaded your example, but I shall and I"ll see if I have anything extra to offer other than what's already been well said.

I had a look at your opening comment, what Resolve does in condensing your speech whisper faster also does.

FFFFF_1.srt - Subtitle Edit 4.0.12.png

It has many options such as repetition and repeat which could be controlling what you're seeing in Resolve but no idea if Resolve is using one of the whisper variants. I do prefer how they condense, I don't want 100% literal transcribing but then you have problems such as your experiencing with text based editing, you need more control.

-Whisper Advanced - extra command line arguments.png

I tried out StoryToolKit AI and it is quite awesome... but is still a bit clunky to work with since I have to tab back and forth between the application and Resolve.

I'm really hoping they just use a better model or at least give us the option to switch models.

it feels just a matter of adjusting the settings to me. Resolve is clearly doing a superb job with transcription, they just need to
1. change the transcribe option to a more "accurate and less nice" method OR
2. give us the chance to change that option via settings

I have a feeling it would be a simple fix.

Meanwhile, I am now using it and I think it saves me tons of time! So, cheers for reminding me of that feature!

True, a threshold of sorts would probably be the fix.

Let's see what they do in the next update. That being said, it's been here since 18.5, so I hope they didn't just put it in and forgot about it.