Generative AI in DaVinci Resolve?

Hello!

I am a filmmaking teacher, and I want to share this with you.

My students are seeing the news about Premiere Pro incorporating new AI tools later this year.

They are mostly independent filmmakers, and because of that, some of those tools are very appealing to them: automatic object removal, clip extension, and the various video image generators.

What we have so far in Premiere Pro is just a announcement. We don't know how well it will work, but we know that their AI tools work very well in Lightroom and Photoshop.

I know we have ways to do some of that in DaVinci, but it's not the same thing, and it's not as easy.

My students are thinking about the inmediate future, and where things are going to go.

We can not deny any more that an "AI-pass" of any film project will be common practice soon, the same way we used to have a "Color Grading Past".

Regardless of controversies, platform wars, and assuming companies do the right thing ethically, I guess my question is:

How will Blackmagicdesign compete with for example Adobe?

Will my students be able to, with a simple click, remove a moving object from their shots?

Will DaVinci manage to do that locally, not sending data to external servers?

I think DaVinci is the superior editor and post platform, but what could I tell my students to give them some confidence?

I'm not just making assumptions. I could tell you about projects I have supervised where AI object removal and clip extension would have had a huge impact in time-saving and budget.

We managed to achieve what needed in Resolve, but it took longer and extra-hours were charged.

Please don't fight.

I am trying to tell you about the point of view of very young independent filmmakers.

Thank you for sharing your ideas.

If they can afford the price Adobe charges because it saves them time, there’s no harm in using Adobe for portions of the VFX work. Blackmagic Design never reveals their future plans so anything stated here will be speculation but I imagine they’ll come up with some interesting tools in the future.

So far, everything DaVinci does with AI is local.

I prefer that.

But maybe generative AI is too complex to be loca.

Also, Adobe promises the user will be able to select between various AI models, all of them external of course.

It seems to me this is unstoppable. Every player in the industry is saying this.

And I want DaVinci to be there, with same or better features, and responsable AI.

Possible?

I think we'll have to wait another five years, regardless of DaVinci or Premiere Pro
Until this all becomes really normal and works quickly and brings significant benefits in practice and not in advertising slogans ))

In addition, you need to understand that Premiere Pro compared to DaVinci is just garbage And this cannot be corrected by any implemented AI

But if DaVinci developers are not fools and do not sleep, but think and act, then this is their golden chance to become the first and only for everything!

That is my hope!

But let's get ready.

Change will happen before we think. Not five years.

I say two years the latest.

It is already a reality in photography.

For context, and this is anecdotal: I have been working on a series of photos lately, experimenting with the AI removal tool in Lightroom. That means: instead of using a clone or a healing tool, I just select a distracting element of the photo and delete it with one click. Works perfect, and fills the inside with the surroundings perfectly.

And let me tell you: after getting used to that, I was back at Capture One, the other photo software, where I have my big collection. Capture One doesn't have this feature, and not being able to easily delete and replace in a photo felt suddenly very strange, and a big limitation.

But, yes, I of course prefer Resolve.

Adobe has a huge catalogue of photos and video available to users for generative AI. No other NLE can offer this.

Ric Marty

Adobe has a huge catalogue of photos and video available to users for generative AI. No other NLE can offer this.

Other general services will come for that; Pixabay and Leonardo, and it's really independent of the NLE virtues too.

AI is in it's infancy: ChatGPT goes all Hal sometimes; Leonardo garbles faces and objects and I understand fast food in the US goes sweet and sour, even when you don't want it.

A lot of AI now surrounds transcription and translation of audio to text; everyone has that. Machine learning and AI used more ingeniously is harder to get right. Where it's really going to make a huge difference for us is in upscaling and NR, and that's already beginning to look better. FX and keying and all manner of 3D composting assist too will be incredibly useful.

Much of this has already been started in Davinci Resolve; they are very much involved in this kind of development. With some quite major advances in that regard from version 18 onwards; for both audio and video.

The problem is and from what you suggest; students and young people are looking for AI to make the films for them. I sincerely hope that doesn't end up being the main case. I know it will one day on You Tube and it will be harder for professionals and creatives to stand out and find their place in this world. That will effect many many artists and technicians everywhere, in every field, one day I fear - we will all be out of a job.

But until then - and I teach young people in a film school too, from time to time - we should urge them to use the tools to think for themselves; independent of the choice of those tools; in my humble opinion

In short; Yes DR will very much have AI as good as anything PP can do. It's free or modestly priced for the Studio; and that's a one off payment for ever; and it's unlikely most students will stick with a subscription cost for PP anyway, in light of that, in the future too.

The main problem for me and many of my colleagues, given that I have been actively working with AI for more than 4 years, is that I am not interested in looking at content in which AI was used
I'm just bored by this
I do this in video and photo production because they pay a lot for it, but I see that the trend is that people are less and less interested in such content, regardless of the level of quality.
I mean content consumers and not content producers

So I think that AI is very interesting in noise reduction, increasing detail and quality, stabilization, and other similar aspects

I'd be much happier if BMD kept its focus on providing core NLE/grading functionality rather than adding "generative" capabilities. There are a growing raft of providers who specialise in generative AI so if that's a need/want then those services can be purchased seperately.

When I'm cooking, I much prefer to use a top quality chef's knife than a swiss army knife.

Focus is important for DPs and post-production software developers

Blackmagic CEO Grant Petty has said before in interviews that he is strongly against Generative AI:

Grant Petty Unleashes on AI
https://www.forbes.com.au/covers/billio ... hes-on-ai/

I don't have a single problem with anything he says. Understand that there's "Machine Learning" tools (which we already have in Resolve, but then there's generative A.I. that's actually training itself from somebody else's artwork or photography on the web, then regurgitating it as a "new" work. I see a lot of moral problems with the latter.

+ 100% with both grant and marc.

like every new advance in technology, they'll closely follow a time when it becomes 'old hat'. way back in time the post house i worked at created a cube with a different video on each face (abacus?), people were astounded, amazed, but pretty soon it looked tired and contrived. now, facebook and insta are swamped with ai images all looking pretty much the same; impossible houses, outlandish fashions, etc., etc., and even the grannies are tiring of it.

and it doesn't matter how much ai steals, it can't (as yet) create something that wasn't stolen in the first place.

I don't use generative AI, but I'm not that radically against it.

We have to separate concepts here.

One thing is how generative AI is being developed today by big corporations, which is frequently abusive.

And a very different thing is generative AI as a tool itself, and what it can do for independent filmmakers.

Imagine something like this: a generative AI tool that works locally in DaVinci, that trains itself only with the media I have in my project.

Why not?

Maybe that wouldn't be enough training data to really deliver good results?

And also, keep in mind: I am not even talking about tools to generate completely new images.

I was always talking about tools to improve and refine existing footage: object removal, object replacement, shot extension.

Let's please not mix it all up.

Blackmagic CEO Grant Petty has said before in interviews that he is strongly against Generative AI:

Grant Petty Unleashes on AI
https://www.forbes.com.au/covers/billio ... hes-on-ai/

I don't have a single problem with anything he says. Understand that there's "Machine Learning" tools (which we already have in Resolve, but then there's generative A.I. that's actually training itself from somebody else's artwork or photography on the web, then regurgitating it as a "new" work. I see a lot of moral problems with the latter.

I might just buy a new BM camera just because of all of this. I need to support more of this.

The amount of people that don't know the difference between real AI and machine learning diffusion garbage is astonishing.

And yes AI should be used for NR, voice isolation, maybe maybe plate extensions (like walls and etc) but not for what some of the people around here are asking for.

And to think that Adobe wants to implement data collection on stuff you do in the projects, like brush strokes and etc to train its diffusion crap is horrifying.

I have two comments:
Are the filmmakers looking for content or a "improve the quality of my video" (like chatgpt does for scientific papers)?

Secondly to comment on the original questions:
"A generative AI tool that works locally in DaVinci, that trains itself only with the media I have in my project."
Improbable because of a lack of source material, overfitting and computational power.
The models require 10 000's of annotated source images, I very highly doubt any common filmmaker has those datasets just laying around. (Yes, I know of GAN and unsupervised models, not relevant here)
As well as not having a GB200 NVL72 just idle waiting for a model.
Training and inference have vastly different computational requirements.

"I was always talking about tools to improve and refine existing footage: object removal, object replacement, shot extension."
Object removal: I can see the inference use there, a better version of the current Object Removal OFX is sorely needed and inpating is a known quantity in single image editing. My worry is that the inpainted area will "flicker" or change frame to frame making it very noticeable.
Object replacement: I'm not sure I understand. Are you talking about removing an object and overwrite the are with a new object? If so, that's a natural evolution of Object Removal.
Shot Extension: Sure. I doubt that it's possible to clone the timeline frames in a way that looks convincing, but if so, yes.


(I misread the reply thread: here's my take on the original questions:

"How will Blackmagicdesign compete with for example Adobe?"

Different markets. Using the strengths of color and ease of editing to create videos faster, Davinci has a sustained advantage for most users. Not including the subscription model issues, which are detrimental to Adobe as it currently stands, see https://www.nytimes.com/2024/06/17/tech ... wsuit.html

"Will my students be able to, with a simple click, remove a moving object from their shots?"

No. At minimum it will take 2 clicks: 1: Selecting the object and 2: clicking the "remove object" button.
Fun aside, it's the solution of "what is behind the object" that is important. The company that solves that to everyone's satisfaction, will have the most users who require that.
Inpainting is a step in the right direction but pattern matching with x frames in either direction is very computationally expensive as well as the constantly updating mask of the object.

Will DaVinci manage to do that locally, not sending data to external servers?

Yes. The parts that need to run locally will run locally. I foresee Blackmagic outsourcing some of the very heavy lifting to cloud companies who have the experience, but I worry about the NDA aspect of having video frames going over the internet.

I think DaVinci is the superior editor and post platform, but what could I tell my students to give them some confidence?

"If it was easy to do it would have been done already."
It totally depends on what your students want to do. If they don't want to edit, then transition into Prompt Engineers.
To get philosophical, art without a soul does not evoke emotions. You need human soul involved to create an engaging final product.
Use the best part of the program, to start generating ideas and storyboards with the fast AI and then evolve the result into a product with the human touch.

I'm not just making assumptions. I could tell you about projects I have supervised where AI object removal and clip extension would have had a huge impact in time-saving and budget.

I don't doubt it. Just be happy that it's not done with film which (i imagine) would drive the costs up even further.
Any tool that makes a task easier will make it cheaper.
Let's play the devil's advocate: Assume that there's a magic button that says "Fix all mistakes" and all mistakes are fixed in 5 minutes. Will any student learn from that? Will the final product look similar/identical to all the other "Fix all mistakes" products? Will film makers learn to look for mistakes during shooting?

We managed to achieve what needed in Resolve, but it took longer and extra-hours were charged.

That's how it goes

Hello!
I think DaVinci is the superior editor and post platform, but what could I tell my students to give them some confidence?

"You will need to learn different pieces of software and multiple techniques on how to do things depending on what you need to accomplish."

I am trying to tell you about the point of view of very young independent filmmakers.

I think in general the views of youngsters is very crooked because of inexperience. They get very excited about amount of features and discussions on who what which is better vs the competition. It's not a totally bad trait. It invokes curiosity and enthousiasm to learn. But it shouldn't cloud creative vision. Software is a tool to assist to create. And some tools can make that process easier / faster. By time they'll learn that preparation, fundamentals and good ideas are much more important than the question of how much effort it takes to fix a mistake or cut corners. And fundamentals can be taught in a generalized way separate from software choice. Once they can see/understand that they'll see that software choice can be slightly less relevant.

A possible hint at what the future may hold.

A Pixys AI or Ursa AI in the future? You never know.

A possible hint at what the future may hold.

A Pixys AI or Ursa AI in the future? You never know.

Like he says 5 minutes in: It's a cloud generated video to video. There's just not anywhere near enough power locally to do the processing.
So actually, if you remove all the fancy it's a laptop sending video to the cloud and receiving it back. I'm sure the specifications would reveal what service they are using (or a packet capture will for sure), but that's left as an exercise to the reader.

The most important thing is that first you need to understand that AI in the form that it now exists - midjourney, sora, adobe, shutterstock and others - is nothing more than blatant theft and development at the expense of other people.
Everyone knows that AI is impossible without stock databases of photos and videos - stock companies blatantly sold databases with people’s work and trained AI on them and didn’t even ask the owners of this content whether they wanted it or not!!!

Just imagine that you have been working for about 15 years and all this time you have been investing in your future and pension fund - and then, at one moment, it turns out that all this was blatantly stolen from you, they stole your income and your investments in the future
And they built a huge amount of money for themselves thanks to your labor...
And right now you have nothing for the moment and for the future!!
This is roughly how it works and this is exactly how things are with AI

And now the question is - are you ready to use the stolen AI capabilities? Essentially stolen from people like you?
And are you ready to believe that AI is a benefit and helps in creating content?

The most important thing is that first you need to understand that AI in the form that it now exists - midjourney, sora, adobe, shutterstock and others - is nothing more than blatant theft and development at the expense of other people.
Everyone knows that AI is impossible without stock databases of photos and videos - stock companies blatantly sold databases with people’s work and trained AI on them and didn’t even ask the owners of this content whether they wanted it or not!!!

Most of the stuff was trained on scraped images from the internet. There's a senate hearing on Ai and copyright.

If you want to hear what corporate slime sounds like listen to the OpenAi guy or the Adobe guy. Especially the "robots.txt" part which is hilariously sad. Or the proposed "opt out" option that basically does nothing since they already stole everything and anything.

I'm sure they will find a way to put a meter on the amount of material used per hundred pixels. I'm sur the money will go to the search engines. Mining for pixels


Ricardo Marty

The amount of garbage spread around on Adobe's TOS is astounding. Although probably understandable as their legal staff always comes out initially with wide range statements that can sound alarming, with pages of limitations following.

How many actually read and comprehend what's left after the limitations? Clearly, not many. Thankfully, they have both issued a revised TOS and given explanations as to what they are specifically talking about.

"Distribution," for instance. That ONLY refers to using the built-in export directly to services options. You could not upload from Premiere to YouTube without allowing specific permission to do that Distribution upload.

Copy and reproduction? When you use any of their services from audio enhancements to image modifications, that by nature is legally copy and reproduction. But YOU are the only one ever hearing or seeing anything because it is not kept on their servers.

As to images for generative AI, that is limited only to submitted images to Adobe Stock *where the submitter specifically checks to allow such use.*

Notice, this means the vast majority of Adobe stock media is not even used.

Not a "fan" of Adobe, as I don't understand emotions attached to massive corporate structures. I just dislike uninformed generalized statements without correct information.

Sent from my SM-S908U using Tapatalk

As to images for generative AI, that is limited only to submitted images to Adobe Stock *where the submitter specifically checks to allow such use.*

Notice, this means the vast majority of Adobe stock media is not even used.

Not a "fan" of Adobe, as I don't understand emotions attached to massive corporate structures. I just dislike uninformed generalized statements without correct information.

Sent from my SM-S908U using Tapatalk

Did Adobe create such things out of thin air? There was no opt in option on Adobe Stock as far as I know. You cannot use the stuff then put the opt in button or use the prior arrangements regarding stock as "you're free to train your crap on my work".

I didn't upload to Adobe Stock but I can attest to Pond5 (aka Shutterstock) where I found myself with a "new sale" of 1.5usd for my entire portfolio for "allow to use in AI training" which was an option that was forcibly checked ON my account. I haven't checked any sort of boxes for AI usage since I created the account or would ever allow. (If the subtle vitriol I have vs generative AI isn't apparent from my post)

Basically they stole my stuff.

What makes you think Adobe did differently? And how could they possibly NOT do it since the requirements for training are enormous.

Have you watched the video? I have and the answers to "how did you train your stuff?" from the Adobe guy are hilariously bad. The OpenAI guy blatantly admits he scraped the whole of the internet and their whole business wouldn't exist without that. And he based it on "robots.txt" (that has absolutely NOTHING to do with AI) and "free for academic research purposes" (ROFL).

/End rant.

And how do you know anything yourself? Decent question.

Among other things, over the last decade, I've learned enough about Adobe practices to know they commission, have teams to create, and buy an enormous amount of imagery both stills and motion.

If you were commissioned to do a program on one of their apps, they can provide any imagery you might need from their in-house supply.

I've asked several staffers who work in this sort of thing how much imagery Adobe has on its servers. Past that it's quite a few terabytes, they haven't a clue. Perhaps the largest in-house reference image library of any company.

And they constantly commission and buy new imagery to add to the pile.

For one job, I did use my own created media I'd made specifically for that job. But they offered to provide all needed media. And clearly would have been more comfortable if I'd used theirs. To use mine, required completing and digitally signing several legal documents, therein attesting to creation and ownership of all contained contents, in order to verify legal use as per copyright laws.

Included in those pages of legal pdfs were clauses they had written, specifying that any media provided by me was purely for use within the specific limitations of this job, and that use of my provided media for this job neither implied nor gave any possibility of use by Adobe outside that job.

It was very detailed.

Sent from my SM-S908U using Tapatalk

[DEV] Using StabilityAI Text-to-Image in DaVinci Resolve and Fusion Studio

https://www.steakunderwater.com/wesuckl ... php?t=6638

And how do you know anything yourself? Decent question.

I said I don't know anything regarding Adobe. I only know what happened to me on Pond5. And based on that and based on what Adobe intends to implement in the future and that video I shared above where the Adobe's guy's answers were corporate slime...I wouldn't put any faith in Adobe's good will.

I'm also quite confident that stuff you sell to Adobe as stock footage , as you said, is for rights of use as stock footage only. Not for training AIs or anything else that's not stock related.

So unless Adobe employees shot all that footage themselves, I really doubt they got approval for this use. The Adobe guy in the video even goes to say that it was trained on footage belonging just to their stock library. There was no mention of getting approval or paying contributors for AI usage since AI wasn't even a thing 2 years ago.

But hey, this is just how I feel about it. I'd be happy to be proven wrong. But...knowing how these models work behind the scenes, the absurd amount of footage you would need to train it to get over the hallucinations and all those other issues would probably bankrupt Adobe if they were to pay stock prices for training.

I'm not such a loyal and gullible fool as many suckers who are deceived by such sites as Adobe, Shattersock, Envato and others
These stocks have been stealing their sales from contributors for many years, this is a known fact!
For example, if a person actually earns from the stock, for example, Shutterstock - $6000 a month, then the stock pays him only up to $2000 best case scenario, the rest is stolen for himself, hiding real sales
(this is already taking into account the interest rate)
In Adobe this is less pronounced but still relevant.
The entire stock business has been on this system for many years.
It is also necessary to take into account that this happens because most of the contributors to the stock are located in backward, poorly developed countries with a poor judicial system. Such a group of countries as the CIS (Russia, Ukraine, Belarus, Kazakhstan, Latvia, etc.) It is the contributors from these countries that make up approximately 80% of all that provide content for all stocks.
The stocks know that these people have no legal protection and are stealing their real income.
And after this, will you believe that AI training is legal?
To do this you need to be either a very naive fool or a completely brainless idiot!!!

We just attended MPTS https://www.mediaproductionshow.com in London and spoke in depth with several guys from the Blackmagic team with some of our feature requests.

They did hint at a delay for the v20 release and the reason for that is that they are working out the credit system for upcoming generative Ai tools, it will likely need to be integrated into the Black Magic Cloud subscription service. But my sense of it was that Blackmagic are a little uncomfortable with the pricing system and are only really looking at including these Ai features because so many people are asking for them.

As well as this "AI set extender." https://www.youtube.com/live/wLw1UU3fxU4?feature=shared&t=9412 they are also trialling a generative extend similar to Premiere. We weren't able to see this in person but this is apparently something potentially coming down the line. They wouldn't disclose the Ai engine behind these new tools (Veo 2/Sora/Runway/Lumalabs) or if they would actually trial them in an upcoming v20 beta but they do already exist.

We just attended MPTS https://www.mediaproductionshow.com in London and spoke in depth with several guys from the Blackmagic team with some of our feature requests.

They did hint at a delay for the v20 release and the reason for that is that they are working out the credit system for upcoming generative Ai tools, it will likely need to be integrated into the Black Magic Cloud subscription service. But my sense of it was that Blackmagic are a little uncomfortable with the pricing system and are only really looking at including these Ai features because so many people are asking for them.

As well as this "AI set extender." https://www.youtube.com/live/wLw1UU3fxU4?feature=shared&t=9412 they are also trialling a generative extend similar to Premiere. We weren't able to see this in person but this is apparently something potentially coming down the line. They wouldn't disclose the Ai engine behind these new tools (Veo 2/Sora/Runway/Lumalabs) or if they would actually trial them in an upcoming v20 beta but they do already exist.

We just attended MPTS https://www.mediaproductionshow.com in London and spoke in depth with several guys from the Blackmagic team with some of our feature requests.

They did hint at a delay for the v20 release and the reason for that is that they are working out the credit system for upcoming generative Ai tools, it will likely need to be integrated into the Black Magic Cloud subscription service. But my sense of it was that Blackmagic are a little uncomfortable with the pricing system and are only really looking at including these Ai features because so many people are asking for them.

As well as this "AI set extender." https://www.youtube.com/live/wLw1UU3fxU4?feature=shared&t=9412 they are also trialling a generative extend similar to Premiere. We weren't able to see this in person but this is apparently something potentially coming down the line. They wouldn't disclose the Ai engine behind these new tools (Veo 2/Sora/Runway/Lumalabs) or if they would actually trial them in an upcoming v20 beta but they do already exist.

I really hope they shove them down a drain pipe. I couldn't care less about the generative garbage. But if they force users that don't need it to pay for it by making it mandatory, it will bite them in the ass...

I've read CEO of Blackmagic said no to generative AI so it will never happen.

I've read CEO of Blackmagic said no to generative AI so it will never happen.

Set extension is generative AI...It added a lamp post as per the prompt...So It already happened.

I've read CEO of Blackmagic said no to generative AI so it will never happen.

Well no local on system generative AI, but server based it is now. He's still making the distinction that Resolve doesn't have GenAI, it's more like a plugin you can pay to use. You would think if they do start charging for Resolve updates as of the release after V20 everyone could get limited credits/month to try the extra AI features for free, pay for credits if they like it, would help quell descent over extra ongoing costs for Resolve.

These are some top level mental gymnastics. There never will be gen AI and if there will be it’s not local so not really BMD thing and if it is local it’s not Resolve but a plugin and if it isn’t a plugin then it isn’t gen AI and if it is, it’s what everyone actually wanted all along, Grant included!

Not sure how ”stealing” in a cloud service is fine but in local machine isn’t but ok.

I have and teach both. Premiere is just a generative toy Ai, additional frame are often waste of time if are not something that I can generate with vector pixel motion. Ask a bit more and do pixel puzzle, luma color inconsistency and more.
No more… all work is for static, they are so far from generate video from text with consistency or accuracy, I do beta and payed for some test but nothing to ready for real production.

I prefer Blackmagic Design statement, local process and security.
If I want to generate videos there are plenty of web oriented ai model to do that, that work better and faster than adobe firefly.
At June adobe will start with new and expansive price for all software and especially for Ai they raise up the price.


Inviato dal mio iPad utilizzando Tapatalk

And how do you know anything yourself? Decent question.

I said I don't know anything regarding Adobe. I only know what happened to me on Pond5. And based on that and based on what Adobe intends to implement in the future and that video I shared above where the Adobe's guy's answers were corporate slime...I wouldn't put any faith in Adobe's good will.

I'm also quite confident that stuff you sell to Adobe as stock footage , as you said, is for rights of use as stock footage only. Not for training AIs or anything else that's not stock related.

So unless Adobe employees shot all that footage themselves, I really doubt they got approval for this use. The Adobe guy in the video even goes to say that it was trained on footage belonging just to their stock library. There was no mention of getting approval or paying contributors for AI usage since AI wasn't even a thing 2 years ago.

But hey, this is just how I feel about it. I'd be happy to be proven wrong. But...knowing how these models work behind the scenes, the absurd amount of footage you would need to train it to get over the hallucinations and all those other issues would probably bankrupt Adobe if they were to pay stock prices for training.

Adobe wrote to all adobe stock contributors years ago about it, they ask if you are allowed to use your stuff for their ai training or not, if you are not agree you could remove all. I’m ones of them, I contribute to adobe stock from years and I rieceved that email and was agree that they can use my shoot for training


Inviato dal mio iPad utilizzando Tapatalk