Free Transcriptions in Resolve using OpenAI Whisper

A few days ago OpenAI released publicly Whisper, their Speech Recognition model which is unlike we've ever seen before, so we created a free tool for Resolve called StoryToolkitAI that basically transcribes Timelines into Subtitle SRTs which can be imported back into Resolve.

Whisper recognizes speech from 97 languages and can translate them into English. So far, we've tried it on footage in English, Spanish, German, and Chinese, and it's really impressive.

StoryToolkitAI can be downloaded for free at the following link (it's written in Python so some knowledge to install it is required): https://github.com/octimot/StoryToolkitAI

Once you install the necessary Python packages on your machine, you can simply go to the timeline in Resolve, press a button in the tool, and wait for the transcription to be processed. When done, you get all the phrases in a JSON file and the subtitles as SRT. Everything is done locally without the need for additional accounts or even an Internet connection, once you have all the packages installed.

Again, this is a free tool that we used in our editing process for almost a week now, but it's quite raw and may be buggy! We may add in the future other learning models like GPT-3 or CLIP to integrate other cool features like content summarization, automated markers etc.

If you're interested in contributing to this open source project, just get in touch!

Thanks for sharing, Octavian. It's much appreciated.
Was only reading about this just the other day, so it's great to see a working model for Resolve out in the wild so soon. Good work.
Cheers
Andy

Thanks Andy!

For now the tool only generates the transcript in a txt file as well as the subtitles in SRT, but the plan is to add more functions soon, like a text UI, speaker recognition and possibly text ideas split into timeline markers.

Again, it's quite amazing how good are the transcriptions and the translations, and that's why we use this exclusively now...

It would be nice to get more feedback from the community to understand what people need in their editing workflows which can be provided by AI / machine learning, and build this open source for everyone to use for free.

I have not had time to test it, but at first glance, it could be very interesting.

Many thanks,
BaGRoS

I tested with a recording in Polish, recorded outside when a fairly strong wind was blowing. Shame it doesn't use my graphics card, but still the end result amazing. Two people speaking, getting into each other's words. The programme only made a mistake once, making a spelling error. However, I, listening to the recording, had to guess, because the word was cut off, which means it can be assumed that it is not an AI error but a video editing error. Next tests with better recorded audio.

I'm waiting for more updates, better use of the graphics card, some kind of specific installer with automatic updating of all components.

MANY THANKS!

Windows 10 here. Do I have to be running Python 3.9? I have 3.10.7 and all dependencies checked out through requirements. But when I run the app.py, it says "Starting GUI" and nothing happens. Any ideas?

Sounds great!
I will check it very soon
Thanks for sharing.

I tested with a recording in Polish, recorded outside when a fairly strong wind was blowing. Shame it doesn't use my graphics card, but still the end result amazing.

I'll push an update to have it select Cuda automatically, if available, in the next days. We ran the code on an old 1070 GPU and the transcription time is around a quarter of the lenght of the audio. Just check back in the next days!

What are your specs btw?

I'm waiting for more updates, better use of the graphics card, some kind of specific installer with automatic updating of all components.

Noted, thanks for the feedback!

Windows 10 here. Do I have to be running Python 3.9? I have 3.10.7 and all dependencies checked out through requirements. But when I run the app.py, it says "Starting GUI" and nothing happens. Any ideas?

Yes, there's a weird bug with Python on Windows 10, but only on some machines (one of our Win machines included). Trying to find a fix...

BTW, although this will not solve the above issue, I recommend running 3.9.13 as this is our production environment. The app works fine in 3.10, but we might add some machine learning libraries that don't support 3.10 yet. Do you know how to install 3.9 and have it run in virtualenv on windows? I recommend pylauncher and virtuallenv.

Laptop Windows11 ; Intel i7 11800k, gtx3080 8GB + 32GB shared memory
64GB RAM, two Samsung 980 PRO PCIe 4.0 M.2 1TB + 2TB
Nothing special but, still smooth.

Many thanks.


Sent from my SM-S908B using Tapatalk

I'm waiting for more updates, better use of the graphics card, some kind of specific installer with automatic updating of all components.

Just pushed an update for CUDA support, but make sure you install cuda torch as per README.

Wow, amazing. I just tested it in a documentary that I'm editing now and it did an excellent job transcribing two old men talking, even very quiet and talking both at the same time.

I could help development when I have time in between jobs. By now I propose a simple add-on, asking for the language. I hard write it because in the first test it interpreted the spoken language as galego instead of spanish.

Thank you very much for this impressive tool. It's going to help a lot in documentary production.

Enviado desde mi Mi 9 Lite mediante Tapatalk

Thanks for sharing!

Wow, amazing. I just tested it in a documentary that I'm editing now and it did an excellent job transcribing two old men talking, even very quiet and talking both at the same time.

I could help development when I have time in between jobs. By now I propose a simple add-on, asking for the language. I hard write it because in the first test it interpreted the spoken language as galego instead of spanish.

I'm glad that this is useful for you too! We're currently transcribing over 96 hours of footage for a doc in 5 languages and it's making really few errors!

The functions definitely need more user input, I'll try to make these available on a very soon update. If you want to contribute, just PM or get in touch directly on Github.

1m44sec movie -> CUDA - 17sec transcribe with medium model.

Next update:
1. Possibility to choose model size from GUI
2. Possibility to choose CUDA / CPU (for some reason torch doesn't want to allocate shared memory so I can only use the LARGE model on the CPU )
3. Perhaps configuring the software in such a way that it allows shared memory allocation at the expense of speed.


Overall, thanks for a super fun time

BaGRoS

1m44sec movie -> CUDA - 17sec transcribe with medium model.

That's significantly better than in our tests. Cool!

Next update:
1. Possibility to choose model size from GUI
2. Possibility to choose CUDA / CPU (for some reason torch doesn't want to allocate shared memory so I can only use the LARGE model on the CPU )
3. Perhaps configuring the software in such a way that it allows shared memory allocation at the expense of speed.

Noted, thanks!
For detailed tweaks, feel free to use the Issues tab in GitHub. Having a centralized place with infos/suggestions would help other users too.

This is incredible, wow! I need to spend some time testing this out.

Do you have a method to import as ranged markers (kinda of like Simon Says?)

This is incredible, wow! I need to spend some time testing this out.

Do you have a method to import as ranged markers (kinda of like Simon Says?)

You can convert subtitle SRT files to ranged markers with this tool:

https://en.editingtools.io/subtitles/

This is incredible, wow! I need to spend some time testing this out.

Do you have a method to import as ranged markers (kinda of like Simon Says?)

You can convert subtitle SRT files to ranged markers with this tool:

https://en.editingtools.io/subtitles/

Whoa thank you!

Nicely done! Thanks for sharing!

Thanks for the great feedback!

Keep it coming, we're really interested in how people approach editing in Resolve and how to develop a tool that is actually helpful.

I'm pushing almost a daily update right now on GitHub, so make sure you're using the latest version. Things are still super raw and maybe buggy here and there, but we use the tool constantly in our editing room at this point. Having these locally generated high quality transcripts really improved our workflow and are changing our editing pipeline.

I'll try to push these updates as soon as possible:
- in tool transcript search
- sync transcript with Resolve playhead
- transcript editing
- markers in Resolve via transcript selection
- automatic opening of transcripts on timeline change
- batch transcription of multiple Timelines
(now you can only render one at a time, although you can queue up transcriptions)

I wish that the Resolve API wasn't so limited in certain aspects to implement some of these ideas faster without trying to hack into it too much, but I guess there's no other option but to work with what we have...

Hi, I've been trying the c++ implementation from ggerganov, and it's transcribing much faster for Intel cpu+amd GPU mac's. It take advantage of multiple cores.

https://github.com/ggerganov/whisper.cpp

Would it be possible to add an option to use this implementation in your story toolkit?


Enviado desde mi Mi 9 Lite mediante Tapatalk

Would it be possible to add an option to use this implementation in your story toolkit?

That's interesting!

I think it would be complicated for people without coding experience to install that, since it's my understanding that you have to compile the code from scratch on your machine.

But if you can think of a way to make that work, let's talk about it on the project GitHub: https://github.com/octimot/StoryToolkitAI

Also, if there are more people interested in this, let me know and maybe we can prioritize it.

FWIW, I just made an update that is really speeding up our editing workflow.

Now, you can navigate the timeline using the transcription via shortcuts, select phrases and mark them directly in Resolve.

https://vimeo.com/759962195/dee07a067a

Thats very nice Octavian, looks like I might have to upgrade my work machine to v18.

Have been playing around with this on an older Intel iMac, and despite a few UI bugs and a couple of setup hiccups, it is a game changer. It 100% nailed an English transcription as well as a Portuguese to English translation + transcription. It struggled with a Indonesian to English translation + transcription, but think that was a tough ask and more a Whisper AI problem rather than this tool.

Will be trying it out again on some M1 machines and will report back.

We’re those problems with Indonesian massive or just a few? I may need that soon, so I’m curious.

We think the problem was perhaps dialect based. We can't be certain that the language being spoken was the same 'Indonesian' that WhisperAI thought it was. It was able to latch on the certain words, but then completely skipped other parts.

Well, it could be mixed with one of the local languages. There is the nationwide "Bahasa Indonesia", which is well standardised, and lots of local languages, which I wouldn't expect to be supported in whole. And then there's Jakartian slang…

So, I'll se how well it works when our project starts.

Well, it could be mixed with one of the local languages. There is the nationwide "Bahasa Indonesia", which is well standardised, and lots of local languages, which I wouldn't expect to be supported in whole. And then there's Jakartian slang…

So, I'll se how well it works when our project starts.

Yeah, this was our thinking. Perhaps a few common words, but otherwise a different language essentially.

I'd love to be able to help in testing. Any tutorials on how to set up and use?
This could be a great alternative to Script Sync used in Media Composer?

I'd love to be able to help in testing. Any tutorials on how to set up and use?
This could be a great alternative to Script Sync used in Media Composer?

Sure, there is a lot of info on the project Github page: https://github.com/octimot/StoryToolkitAI

For installation instructions, go here: https://github.com/octimot/StoryToolkit ... LLATION.md

We're currently working on an easier way to get the app running for folks who aren't much into python/coding/etc..

On this page, we'll slowly add releases of the tool for different platforms: https://github.com/octimot/StoryToolkitAI/releases

Right now, Mac M1, Mac Intel and Windows (with CUDA GPUs) versions are available.

Hello there,

I'll check that this week... I'll report here on whatever may be interesting...
Thank you anyway to make it available so far.

We're currently working on an easier way to get the app running for folks who aren't much into python/coding/etc..

On this page, we'll slowly add releases of the tool for different platforms: https://github.com/octimot/StoryToolkitAI/releases

Right now, a Mac M1 and Mac Intel version is available.

that will be awesome for us folks that have no knowledge of python etc

that will be awesome for us folks that have no knowledge of python etc

Have you tried the standalone version for MacOS? You only need to download it, install homebrew and ffmpeg with cca. 2-3 commands (if all works fine) and it should work.

I recently tried it myself and it seems pretty up there. There were a few errors, especially when I was the speaker. That was expected though as I tend to stutter and stumble through my words, which would lead the program to get confused. It ended up processing 220 minutes of audio in 26 minutes via cuda though on the medium model which is impressive.
I am having a lot of trouble getting it to work with Davinci studio on Windows, but I think that is a user issue as I am very new to Python. As in only got it for this new.
One thing I would like to see with the program, and I'm not sure if it has it yet or not, is to have the functionality of adding parts of the transcript to Davinci's title editor to place and modify how we see fit. I suppose we can kind of do that already though by matching the title's length up with the markers from the transcript.
Something else I would love to see, again it might already be there, is for the transcript to designate who is saying what. I'm already trying that by adding (Person 1) and (Person 2) into the prompt and I'll update later if it works. I'm also testing out the large model to see how and if it differs but that'll take significantly longer as I have to us my cpu because it needs 20 gig of memory and my gpu maxes out at 8.

Overall, I like this program and would probably love it once I can get it to talk with Davinci.

EDIT: It now works with Davinci thanks to Octavian's help. My speaker test didn't work though and I still don't know how the large model differs as it took too long for me.

Overall, I like this program

Thanks for the feedback!

The speaker recognition is on the works and listed as a future feature on the project page: https://github.com/octimot/StoryToolkitAI#key-features

There's actually a lot of stuff that you can do now from the tool once you have the timelines transcribed, like searching for stuff and taking the resolve playhead to the position of your text, marking different blocks of text in Resolve etc.

The idea is to optimize the editing process using AI, so imho there are some old editing habits that might not be necessary once everything is up and running - and that's why we're prioritizing the integration of AI based features (semantic search, summarization, hopefully visual recognition via CLIP etc.) over some stuff that folks are used to (like speaker recognition etc.)

Feel free to ask stuff or add feature requests on the project issues page: https://github.com/octimot/StoryToolkitAI/issues

This way we'll keep the Blackmagic forums noise free and keep all the ideas in one place.

Octavian,
Thanks so much for developing and sharing this. I am not on GitHub yet but I will be creating an account soon. However, I have one suggestion for your installation instructions...

Change #7 (and at the very end) in the Windows installation to venv\Scripts\activate instead of activate.bat

What I found is...
"Just run activate, without an extension, so the right file will get used regardless of whether you're using cmd.exe or PowerShell."

Or, maybe it is my mistake that I was using PowerShell but that's what I always use. Thanks again for everything!

Edit: Never mind, it worked the first time and now isn't working so I guess the good ol' command prompt and venv\Scripts\activate.bat is the way to go! Sorry!

I think I'm gonna give it a try.
I already have Whisper installed in a virtual environment (on windows), everything is working.

Doing a : pip install git+https://github.com/openai/whisper.git - in my venv should be just fine right? (and install the package required if there is more than those already installed for Whisper)

Btw : https://github.com/jianfch/stable-ts << someone made a script for word level timestamps, works with Whisper.

Edit:
You should check out this repo : https://github.com/antiboredom/videogrep

Code: Select all

Videogrep is a command line tool that searches through dialog in video files and makes supercuts based on what it finds. It will recognize .srt or .vtt subtitle tracks, or transcriptions that can be generated with vosk, pocketsphinx, and other tools.

If your tool had a "side" script like this, that could create "regular" EDL files too, because this one create a particular flavor EDL file... it doesn't open in Resolve. It can export XML files too, but imported as timeline, video and audio is not linked and it's a pain to use.

I say that because it's a super useful tool (at least for me). Find and extract the only part I need, based on what it said, on long videos.. it's pretty cool.

Edit:
I'm not a dev, but would it be better to have the logs and conf files in the same directory as the github install? So the files would not be scattered in different folders (that's one thing I dislike with some application, when they put tings in many different folders, appdata, "user" folder, roaming and local.

I installed in my Whisper venv :

Code: Select all

pkg_resources.VersionConflict: (typing-extensions 4.4.0 (c:\tools\ai\whisper\lib\site-packages), Requirement.parse('typing_extensions==4.3.0'))

< It looks like when Whisper is installed first alone, it installs a newer version of typing-extensions

Same for pkg_resources.VersionConflict: (tokenizers 0.13.1 (c:\tools\ai\whisper\lib\site-packages), Requirement.parse('tokenizers==0.12.1'))

Edit 2:
I installed the correct version of these two, the script starts (no popup of the app appears) .. I see

Code: Select all

INFO: Running StoryToolkitAI version 0.16.18
INFO: Using cuda for Torch / Whisper.

, then it stops after 10 secs

(For info, I have Python 3.10 installed)

@Videoneth

I'll look into the stuff you mentioned, thanks! There's a lot of stuff that can be extended and we're constantly building it up also based on our editing needs.

For eg., I just pushed a really cool feature which searches your transcripts by concepts and meaning using AI. It's raw, but already does a pretty good job and it runs locally on your machine.

Regarding installing on windows, please see the detailed installation instructions here: https://github.com/octimot/StoryToolkit ... md#windows

Anything not per instructions is difficult to predict (for eg. Python 3.10 instead of 3.9) because we're using some experimental packages which are quite picky about their dependencies. But once you get it working, it really works.

If you still have problems installing, feel free to write here https://github.com/octimot/StoryToolkitAI/issues/9

Yeah I tried this way first because I have so many "AI" python app, in different environment, with always these big different version of pytorch packages taking so much space.

Btw, can "app.py" take the "--model_dir" arg so when it passes it to Whisper, then it doesn't download again the model (I have the medium and small ones in a specific folder, I like having these files in the same places ).

I really often have to search specific things in 200-400 videos files (which have each a .str file assoticated to them, and when they don't, I use Whisper for it.. but it's manual for each file, it would be create to have something ( ) that scan a folder and transcribe the that doesn't have any vtt or srt associated to them lol)

Btw, can "app.py" take the "--model_dir" arg so when it passes it to Whisper, then it doesn't download again the model (I have the medium and small ones in a specific folder, I like having these files in the same places ).

No, it's using the default cache folder, since whisper is not the only package in the tool that requires a model. You could probably create symlinks in the cache folder to the folders where you have your models, but that's a bit impractical... The reason these folders are standardized is because it makes it easy for people to simply run a script without dealing with downloads, verifications etc.

I really often have to search specific things in 200-400 videos files (which have each a .str file assoticated to them, and when they don't, I use Whisper for it.. but it's manual for each file, it would be create to have something ( ) that scan a folder and transcribe the that doesn't have any vtt or srt associated to them lol)

You can already batch transcribe multiple audio files by simply turning on the tool without Resolve and selecting multiple audio files for transcription.

Searching a folder of transcripts is coming in a future update. Again, I'm talking about semantic search, not just finding words - the "advanced search" that we have now for individual transcripts

Well, I've created a new virtual environnement, with 3.9.13, followed the installation process describe on github et and I get a module error : ModuleNotFoundError: No module named 'tkinter' -

The official Python installer should have that module - see instructions on where to download it.

Let's move this conversation in the windows installation issue page here: https://github.com/octimot/StoryToolkitAI/issues/9

This way we can keep the bm forum free of noise and help others install the tool if they run into similar problems!

Well, I've created a new virtual environnement, with 3.9.13, followed the installation process describe on github et and I get a module error : ModuleNotFoundError: No module named 'tkinter' -

The official Python installer should have that module - see instructions on where to download it.

Let's move this conversation in the windows installation issue page here: https://github.com/octimot/StoryToolkitAI/issues/9

This way we can keep the bm forum free of noise and help others install the tool if they run into similar problems!

I deleted my post because it was an user error! I unchecked the "tk" box while installing the 3.9 by mistake, went to fast during the installation lol

Btw, it works

The main window needs a scrollbar
It works, I'm testing now.

I really think an option should be added to specify a folder where everything gets downloaded (and used). And the config files too. Just a suggestion.

I used another tool, from a git repo, and everything is contained in one place, all the different model for upscale, face restoration, etc. from all the different sources the app uses.

Loading an srt file would be great too.

But good work man for this

ps.
where is the all-mpnet-base-v2 downloaded? I saw that it downloaded 458M when I tested the advanced search
Edit.. found it.. in USERS/.cache/torch/sentence_transformers

Yep, it would definitively be a + to get everything at one place. I think I'm gonna put a symlink there in the mean time

when we do a search, having an index generated with the relevant sentences and timestamps would be great too, easier to navigate after a serach.

Thanks for the cool free software: i did not manage to integrate it with my DVR Studio (using python 3.10 on Windows 11 Pro) but it works transcribing the audios from my videos from romanian to youtube subtitles !
Awesome !

Pretty amazing! I'm using the standalone on Mac. The first thing I was testing was Chinese. I'm not impressed with that English translation (I have existing, human-made subs to compare it to), but it did a perfect job on some English, French, Spanish, and even Nepali!

It *does* in fact hallucinate when it encounters silence for a long time. I accidentally gave it a timeline that was missing audio, and it came up with this:

-------------

I have attached my sunflower seeds to possession.
So what do you say?
I'm fine, thank you.
Sure, I have a sunflower seed.
Let's go out to see it.

--------------

Thanks for building this tool!

OMG, AI poetry from silence!