Artificial Intelligence A.I. to smoothen Jump Cuts

  • Author
  • Message
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 12:17 am

For jarring Jump Cuts, as a solution, is it feasible for an NLE software like Davinci to use A.I. to create a blurring effect that is only visible at the frame-level. This blurring effect could be like a morph effect.

What I'm saying is that A.I. could be used to create a blur between the two positions of the person's body, and use a color scheme in that blur that is taken from the color scheme of the person.

I imagine that it would be largely invisible to the human eye because it happens so fast.

But if it were slowed down into slow motion, it would resemble the VFX seen movies where a body transforms into particles.

What I'm suggesting is that A.I. is used to add a color-matched blur between the two positions of the person, at the start and end of the jump-cut, so that when viewed at normal frame rates, the eye does not see a jump cut, but rather a blur that is so fast that it is more visually acceptable than a jump cut.

I realise the "Smooth Cut" already does something like this, but I'm proposing a more radical effect of blurring that can smoothen much more jarring jump cuts. The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.
Offline

Peter Cave

  • Posts: 4199
  • Joined: Thu Aug 23, 2012 6:45 am
  • Location: Melbourne, Australia

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 3:12 am

Have you seen this transition effect anywhere at all?
Resolve 19.0.3 Mac OSX 15.1 Sequoia, Monitor 3G, FSI SDI grading monitor.
Mac M1 Studio Max 32GB
Offline

sjaehnert

  • Posts: 62
  • Joined: Thu Mar 14, 2019 12:41 am
  • Real Name: Steven Jaehnert

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 3:32 am

two cents: I haven’t been able to use the smooth cut in a way that’s worked for me yet. That said in the interview I tried it on last, the subject was outdoors, moved his face slightly and the person sitting in the background got up and walked behind the subject.


Sent from my iPhone using Tapatalk
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 4:39 am

Peter Cave wrote:Have you seen this transition effect anywhere at all?


No, but I thinking that such an effect could be perceived by the viewer as the person moving from one position to the other - rather than the jump cut that looks like a glitch.

Perhaps the A.I. could review the deleted footage, and mathematically create a transition (from the original footage) to simulate a clean transition, instead of a jump cut.

I thought of these ideas because I'm creating studio videos where the speaker is standing while speaking into the camera, hence, there is more lateral body movement and more hand/arm motion, compared to a seated speaker. Hence, jump cuts don't look good.
Offline

Rick van den Berg

  • Posts: 1470
  • Joined: Tue Jun 02, 2015 7:47 am
  • Location: Netherlands

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 9:09 am

this might be the most difficult feature request i've ever seen.

also one of those things where i'm like, this should not have any priority at all. but it could be fun.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 14, 2024 5:36 am

I'd like to re-visit this suggestion I made in 2019. Back then, AI hadn't really become to flavor-of-the-month as it is now. Now in 2024 we are seeing OpenAI's Sora creating entire videos.

Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 12:44 am

rsf123 wrote:Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?

Have you tried Smooth Cut? Done in the right way, it can work very well.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Peter Chamberlain

Blackmagic Design

  • Posts: 14447
  • Joined: Wed Aug 22, 2012 7:08 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 1:08 am

rsf123 wrote:For jarring Jump Cuts, ......The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.



try adjusting the smooth cut duration.
DaVinci Resolve Product Manager
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 4:47 am

Marc Wielage wrote:
rsf123 wrote:Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?

Have you tried Smooth Cut? Done in the right way, it can work very well.


Peter Chamberlain wrote:
rsf123 wrote:For jarring Jump Cuts, ......The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.

try adjusting the smooth cut duration.



I do use Smooth Cut, but it is no use when the person being filmed waves arms around, such that the hands and arms are in very different positions before and after the cut. That's where I hope A.I. can bridge the gap where Smooth Cut cannot.
Offline

Andy Mees

  • Posts: 3575
  • Joined: Wed Aug 22, 2012 7:48 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 11:31 am

Based only on the premise that such a feature must surely be so very very very far down the list of the things that actually need doing/implementing, and that all those other things must be addressed first, I would gladly give this a hesitant +1. ;)
Offline

Jim Simon

  • Posts: 33150
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 8:52 pm

rsf123 wrote:Smooth Cut, but it is no use when the person being filmed waves arms around,
That's when you cut to B-roll. ;)
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 2:01 am

Jim Simon wrote:That's when you cut to B-roll. ;)

I just fell down laughing because I said the exact same thing at the same time. There's an art to using B-roll footage to bridge difficult edit points.

I'll point out (as I've done before) there was for decades a "BBC Rule" that said if you juxtaposed two different sentences from an interview subject, you couldn't hide the edit and had to just show the jump cut, so that nobody would be accused of obviously trying to re-edit their words to make them say something they never said. For example, if they said, "I am not guilty," and then you eliminated the word "not" and just used Smooth Cut to cover up the gap... now they're saying "I am guilty!" The morality of editing in interviews has to be a consideration.

In the United States, most news organizations just try to use good judgement and never go that far, but they are sometimes guilty of applying a certain "slant" to news stories: like using the takes where somebody is frazzled or awkward, or they're fumbling for the right words. When I was in film school and studied documentary filmmaking, these were hotly-debated topics.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Dermot Shane

  • Posts: 2818
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 5:51 am

sjaehnert wrote:two cents: I haven’t been able to use the smooth cut in a way that’s worked for me yet. That said in the interview I tried it on last, the subject was outdoors, moved his face slightly and the person sitting in the background got up and walked behind the subject.


Sent from my iPhone using Tapatalk


i did a similar shot a few years ago, background movement on the B side of the transition made smooth cut unworkable
i roto's the face on both sides of the cut (this was before magic mask)
did a smooth cut on the face only
tracked and comped the new / morphed face on the A side of the shot
was faster than pulling plates and sending to VFX, but sending would have been the better answer had it been flagged eariler
this was in a feature for a studio, the shot passed all QC
Offline

ricardo marty

  • Posts: 1698
  • Joined: Wed Apr 26, 2017 4:03 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 6:16 am

Sometimes you can increase or decrease the size of the shot your cutting to and in certain contents you could put the dip to color effect.

Ricardo Marty
DVR_Studio 19.5. Asus ProArt PD5, 2.5 GHz i7 16-Core 64GB of 3200 MHz DDR4 RAM. Lenovo Yoga slim 7x Snapdragon 78-100 12 core elite 32gb ram adreno gpu, 500gb nvme M.2 ssd. Win 11. BenQ PD3420Q, Sony FS700R, Bmp4k, Sony A6700. PreSonus AudioBox
Offline

ricardo marty

  • Posts: 1698
  • Joined: Wed Apr 26, 2017 4:03 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 6:16 am

Sometimes you can increase or decrease the size of the shot your cutting to and in certain contents you could put the dip to color effect.

Ricardo Marty
DVR_Studio 19.5. Asus ProArt PD5, 2.5 GHz i7 16-Core 64GB of 3200 MHz DDR4 RAM. Lenovo Yoga slim 7x Snapdragon 78-100 12 core elite 32gb ram adreno gpu, 500gb nvme M.2 ssd. Win 11. BenQ PD3420Q, Sony FS700R, Bmp4k, Sony A6700. PreSonus AudioBox
Offline

Steve Fishwick

  • Posts: 1309
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 10:39 am

Jim Simon wrote:That's when you cut to B-roll.


As Jim implies; AI should never replace the good old fashioned c/away. Maybe I'm old fashioned but in over 35 years in the cutting room and increasingly having to deal with 'spray and pray' hand held reality rushes; there has never been a time I had to leave a jump cut in (except for effect in a music sequence perhaps), because there was no other solution; no matter how bad the rushes - there is always a more elegant solution.

When I shoot my own footage; because I am an editor in background, I will shoot interviews on sticks and then ask the question 3 times; each with either a MLS, MS or MCU/CU, so I can cut in and out with impunity, for emphasis; even the best camera person can miss the emotional moment by zooming in too late. Whereas, even when they don't answer exactly the same each time, you can always find the cut in, at the right point.

Obviously with 4K and plus you can just reframe too. Cutting in on the same angle used to be known here as the 'American Cut', I don't know why and many still dislike it, I don't, quite the opposite often, as long as the angle changes sufficiently.
Offline

Jim Simon

  • Posts: 33150
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 1:05 am

Steve Fishwick wrote:there has never been a time I had to leave a jump cut in
Dude, I'm beggin' you...OFFER A COURSE FOR YOUTUBERS! ;)

90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 5:30 am

Jim Simon wrote:
rsf123 wrote:Smooth Cut, but it is no use when the person being filmed waves arms around,
That's when you cut to B-roll. ;)


For my videos, they're like interviews with a person where you don't go jumping to B-roll. Moreover, the cuts are quite frequent, so it'd look funny with numerous jumps to B-roll.

In the era of DALL-E and SORA that can create videos of everything, it might not be too big a hill to climb to just fill in the gaps between a Jump Cut.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 6:38 am

Jim Simon wrote:90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".

Yeah, a lot of that is distracting and bizarre. I get that it's a "style," but it doesn't exactly look real and natural to me.

Two of the things we often did in local TV news: we'd get a reverse angle of the interview subject from the back while the interviewer was nodding and/or taking notes. Perfect for B-roll when you had to (say) cut a minute out of the middle of an interview and juxtapose two sentences together just to get to the point.

And another was the (somewhat amateurish) "shot of the interviewee's hands moving," without seeing their face in the shot. Not exactly smooth, but for some it's better than a jump cut.

The third thing I often see on the big news shows (60 Minutes, 20/20, etc.) is going to the "Killer Wide Shot" from the back of the room that shows the lighting setup, the cameras, the sound guy, the director, and so on. If it's far back enough, you can't see the people's mouths enough to worry about lip sync. So that'd be a worst-case scenario of a 3rd angle to fall back on.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Steve Fishwick

  • Posts: 1309
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 9:05 am

Jim Simon wrote:Dude, I'm beggin' you...OFFER A COURSE FOR YOUTUBERS!

90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".


:lol: Thanks Jim but I fear it would be wasted; though I've thought about it and do teach from time in colleges; and I agree with you totally!
Marc Wielage wrote:Yeah, a lot of that is distracting and bizarre. I get that it's a "style," but it doesn't exactly look real and natural to me.

Two of the things we often did in local TV news: we'd get a reverse angle of the interview subject from the back while the interviewer was nodding and/or taking notes. Perfect for B-roll when you had to (say) cut a minute out of the middle of an interview and juxtapose two sentences together just to get to the point.

And another was the (somewhat amateurish) "shot of the interviewee's hands moving," without seeing their face in the shot. Not exactly smooth, but for some it's better than a jump cut.

The third thing I often see on the big news shows (60 Minutes, 20/20, etc.) is going to the "Killer Wide Shot" from the back of the room that shows the lighting setup, the cameras, the sound guy, the director, and so on. If it's far back enough, you can't see the people's mouths enough to worry about lip sync. So that'd be a worst-case scenario of a 3rd angle to fall back on.


Yes Marc, all witnessed and experienced here too - Oh God, the meaningless hand shot they used to take :lol:

If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Most good camera people now will be more creative and purposeful with their B-roll. It could be family photos in the room, even a shot out of the window etc. Often there is a second camera from the side too. Years ago there was a brilliant filmmaker here, I forget his name, who made a film about the Everly brothers; and they being twins he shot Phil straight on and Don side on, and it was a very powerful interview.

If you think about it, and this is what I do teach; editing is the most cinematic of all cinema's elements: Photography, sound, music etc. all come from elsewhere; but the pure juxtaposition of shots, is unique to the moving image. And the reaction shot can be the most cinematic of all.

If there is a conversation between two or more people, it is not always the person talking who is the point of interest; as in life. The other person's reaction might tell the story better.

And the example I always use is from The Hustler, cut by the brilliant Dede Allen; where George C. Scott is haranguing Paul Newman, Fast Eddie, telling him he's finished; Everyone is just looking at the floor embarrassed; and it's all a series of reaction shots.
rsf123 wrote:For my videos, they're like interviews with a person where you don't go jumping to B-roll. Moreover, the cuts are quite frequent, so it'd look funny with numerous jumps to B-roll.

In the era of DALL-E and SORA that can create videos of everything, it might not be too big a hill to climb to just fill in the gaps between a Jump Cut.


My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar. Though I will add this, if it's absolutely necessary (Court legal Video may be an instance) then it is much better to leave it as a jump-cut; since in my mind some kind of attempt to be invisible, is dishonest - you are manipulating what was said rather than acknowledging editorial choice.

One day we'll be able to just sit back like drones and let AI do everything; and have to do nothing creative ever again. I love AI as an aid but I never ever want it to cut for me.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 1:44 pm

Steve Fishwick wrote:[

My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar.


Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.

The videos I make are talking head videos where the subject matter is highly technical, tertiary level, postgraduate subject matter. Hence, during filming, the person often tries several takes to explain the same point. Then, during editing, we find the clip which was the best way of explaining, and then have to splice it together with other parts of the filming to produce what appears to the viewer as a seamless single presentation.

There are no B-roll. There are diagrams inserted into the video, but always with the speaker visible, because the speaker points to the diagrams to explain them.

In this scenario, I try the best to merge portions of videos into a seamless presentation. The hardest parts are where the ends being joined consist of the speaker having vastly different hand gestures.

That is why I thought AI might come to the rescue.
Offline

ShaheedMalik

  • Posts: 1171
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 2:57 pm

rsf123 wrote:
Steve Fishwick wrote:[

My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar.


Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.

The videos I make are talking head videos where the subject matter is highly technical, tertiary level, postgraduate subject matter. Hence, during filming, the person often tries several takes to explain the same point. Then, during editing, we find the clip which was the best way of explaining, and then have to splice it together with other parts of the filming to produce what appears to the viewer as a seamless single presentation.

There are no B-roll. There are diagrams inserted into the video, but always with the speaker visible, because the speaker points to the diagrams to explain them.

In this scenario, I try the best to merge portions of videos into a seamless presentation. The hardest parts are where the ends being joined consist of the speaker having vastly different hand gestures.

That is why I thought AI might come to the rescue.


Using a second camera or B-Roll would be still the best solution. Even with those diagrams, you can shoot the diagrams as secondary shot. Then back to shot of the speaking pointing to them.

Take a look at the Multicam Editing tutorial.

https://www.blackmagicdesign.com/produc ... e/training
Offline

Steve Fishwick

  • Posts: 1309
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 3:48 pm

rsf123 wrote:Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.


I'm Sorry but you can hardly blame me for that; not knowing your particular example. As Shaheed suggests, you can use the diagram inserts as full frame shots.

If you'll forgive me whilst I may surmise you are extremely knowledgeable in your field, you tend to suggest you are very inexperienced in filmmaking and I would therefore urge you, rather than just dismiss my examples, seek out other examples more relevant, of how such material may be treated and approached, IMV more properly. So when the speaker moves to indicate something on the diagrams, the logic of cutting follows, in that their hand or pointer will be in shot too, for example, and your intended audience, will not be confused or jarred by an unnecessary jump-cut.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 2:59 am

Steve Fishwick wrote:If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Well, there's no harm in shooting a big wide reverse with just a light and a camera in the frame, with the back of the interviewee's body to camera. At least it's coverage done on the day of the interview, even if there's technically nobody else sitting in the chair. Often, it's just a segment producer or somebody else who doesn't necessarily want to be seen on camera -- no biggie.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline
User avatar

Charles Bennett

  • Posts: 7046
  • Joined: Sat Nov 05, 2016 11:55 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:16 am

What really annoys me on jump cuts is not so much the picture but the clipping of the front and end of the dialogue removing the natural flow of speech. This is a growing trend amongst YouTubers to the point where some videos are just unwatchable.
Resolve Studio 19.0.3 build 5
Dell XPS 8700 i7-4790, 24GB RAM, 2 x Evo 860 SSDs, GTX1060/6GB (561.09 Studio Driver), Win10 Home (22H2), Speed Editor, Faderport mk1, Calibrated Eizo ColorEdge CS230 + BenQ GW2270 + Samsung SA200, Canon C100mk2, Zoom H2n.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 2:29 pm

Marc Wielage wrote:
Steve Fishwick wrote:If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Well, there's no harm in shooting a big wide reverse with just a light and a camera in the frame, with the back of the interviewee's body to camera. At least it's coverage done on the day of the interview, even if there's technically nobody else sitting in the chair. Often, it's just a segment producer or somebody else who doesn't necessarily want to be seen on camera -- no biggie.


In these days of pandemic zoom calls, where everything in frame looks immaculate, but all the garbage in the room is shoved out of camera view? Well, it's the same with many smaller studios, particularly home studios. Virtually all the suggestions made in the thread pertain to professional studios. I can't even fit a 2nd camera for angle shots in the room that functions as my studio. But no one needs to know that from the video image with a nice background.
Offline

Jim Simon

  • Posts: 33150
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:03 pm

Charles Bennett wrote:What really annoys me on jump cuts is not so much the picture
That is the most annoying part. It's very visually distracting.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Jim Simon

  • Posts: 33150
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:08 pm

rsf123 wrote:I can't even fit a 2nd camera
You don't necessarily need multiple cameras.

You just need ONE GOOD TAKE.

If that requires a dozen tries, do them.

Because to paraphrase Gene Krantz..."Jump Cuts are NOT AN OPTION!" ;)
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 11:45 pm

Jim Simon wrote:
rsf123 wrote:I can't even fit a 2nd camera
You don't necessarily need multiple cameras. You just need ONE GOOD TAKE. If that requires a dozen tries, do them. Because to paraphrase Gene Krantz..."Jump Cuts are NOT AN OPTION!" ;)

It's not always possible to do multiple takes, particularly on an interview. I'm reminded of a situation on the Red User Forum about 8-9 years ago where somebody was interviewing an Olympic athlete on the verge of a nervous breakdown, and they dissolved into tears on camera while telling a harrowing story of their personal grief that kept them off the Olympic team. And just at that moment, the Red camera overheated... and the fan blew so loud it obscured her sobs.

Granted, documentaries and Zoom calls are kind of a special case, but even there, it's possible to shoot coverage. One thing we tell clients is to shoot a reaction shot of the person watching the zoom call (like an over-the-shoulder), and that might be a good moment to jump to the next answer. A skilled editor can make it work without straining Smooth Cut. And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 12:37 am

Marc Wielage wrote: And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.


Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.
Offline

Videoneth

  • Posts: 2175
  • Joined: Fri Nov 13, 2020 11:03 pm
  • Real Name: Maxwell Allington

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 12:22 pm

"AI" is not "magic". :D

If training a model to specifically make the "smooth cut" transition better-looking, why not?

I use the "smooth cut" very often, when the only things that really move a little bit are the eyes, mouth, things like that. I don't care too much with things jumping out in the background, it is what it is, there is no way around it when it's the same scene.

I don't like the morphing effect but it can be use for artistic purpose (which I did in the past).

But it's such an elaborate process for a tiny, tiny problem that doesn't need to be corrected imo.

I personally don't care about jump cuts; they are totally fine (except when it's to remove silence after every single words). But craming b-rolls everywhere just for the sake of hidding everything is worse imo.

I've seen those, lol. Like, dude, you're making a tutorial; don't try to make something cinematic (or whatever) for this 8-minute video. Adding fluff that adds no value to anything is more annoying than removing something with no value like silence (which can become a jump cut).

I think the OP forgot to think about the audio part. Most jump cuts are there because words or silences were removed. 

Imagine you have someone with their hand up in the air, and the hand is slowly raised before the cut. Then the hand is on the bottom of the frame. How does that work? You add a "smooth-AI - Cut" between the clips, a 2–5 sec. transition, so it would look more natural and not as if the person is slaming their hand.

The "AI" would analyze the two frames and create all the elements in-between, and assuming the result is good, you'll end up with a gap for the audio that needs to be filled.

At the end of the day, audio is 10 times more important than video anyway. People can watch the most compressed, out-of-focus video with very good audio. But they will 100% complain if the images are pixel-perfect with crappy audio.

You can "zone-out" while watching a video with a tons of jump cuts (like the earlier videos of Mr Beast, one example that comes to mind, I couldn't watch them years ago because they "stressed" me out lol), and still have a good experience because the audio is good. The opposite doesn't work well 100% of the time.
Windows 10
v19.0.3
nVidia 3090 - Studio 561.09
Offline

ShaheedMalik

  • Posts: 1171
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 1:37 pm

rsf123 wrote:
Marc Wielage wrote: And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.


Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.


You if you have to put a jump cut in crucial explanation, then it is a poor explanation and you have to redo it anyway. You want AI to make up for poor filmmaking.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 3:48 am

ShaheedMalik wrote: You want AI to make up for poor filmmaking.


Actually, yes. In an ideal world, the speaker will roll out pearls of wisdom in their speech with the ultimately-good version being the first and only take. But in the real world, we do make errors, and require many takes/tries/attempts. And sometimes, the good stuff is spread across a few takes, and needs to be stitched together to give an appearance of being seamless.

So, yes, I want A.I. to make up for poor filmmaking.
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 3:54 am

Videoneth wrote: The "AI" would analyze the two frames and create all the elements in-between, and assuming the result is good, you'll end up with a gap for the audio that needs to be filled.


No. Instead, I envisage that A.I. can offer to overwrite the portion of your video, before and after the cut. Hence, there would be no break in the audio.

Example:

Clip 1, the person says, with arms spread wide apart, "I caught a massive fish and it was // ... er, what type of fish, hmm, I forget, I think .."

Clip 2:, the person says, with arms by their side, "and it was // an Atlantic Salmon".

If you cut out the part where he forgot his lines, you end up with a sudden jump where the arms are spread-eagled apart ... suddenly cut to arms being by his side.

So the AI would re-create and REPLACE the video on either side of the cut, so that the AI footage would show the arms coming together. And this movement of the arms can be deduced from the other footage where the arms go from position 1 to position 2.

Hence, by the AI creating and replacing footage either side of the cut, it gives a smooth arm movement with no cut.

No gap in the edited audio, ""I caught a massive fish and it was an Atlantic Salmon."

I have been experimenting with written-prompts that instructs ChatGPT OpenAI how to create diagrams, and I feel that the above is well within the capability of OpenAI's algorithms.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 6:34 am

rsf123 wrote:Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

Well, the B-roll cutawway will always work. If the guy says, "I caught a fish this big," and we cut to a still of an enormous fish and then back to him, the audience will get it. A lot depends on your own imagination and the available stock footage (or still frames or whatever you have).

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Sure you can, if the audio continues underneath. This is a documentary technique that goes back almost 100 years, literally to the beginning of sound films

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.

Sure, but I'm kind of morally opposed to the amount of fakery in OpenAI imaging, especially if it's used to bend a human being into saying or doing something he or she didn't actually say or do. There are uses of AI that I'm OK with: for example, if you released a foreign version of a Hollywood film, and you wanted the American actors' lips to move along with the foreign-language dialogue. That's been demonstrated and it can work. Morally speaking, I have no problem with it. I do have a problem with AI if it's stealing pieces of somebody else's creative work, or if it's putting human artists (or actors or technicians) out of work.

I'd point to the recent lawsuit between actress Scarlett Johansson and the Chat-GPT people as an example of AI gone horribly wrong...

Scarlett Johansson told OpenAI not to use her voice
https://www.theverge.com/2024/5/20/2416 ... gal-action
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 9:26 am

Marc Wielage wrote: I'm kind of morally opposed to the amount of fakery in OpenAI imaging, especially if it's used to bend a human being into saying or doing something he or she didn't actually say or do. There are uses of AI that I'm OK with: for example, if you released a foreign version of a Hollywood film, and you wanted the American actors' lips to move along with the foreign-language dialogue. That's been demonstrated and it can work. Morally speaking, I have no problem with it. I do have a problem with AI if it's stealing pieces of somebody else's creative work, or if it's putting human artists (or actors or technicians) out of work.


In terms of degree of artificiality, one could say that fabricating the movement of a person's lips to fake them talking in a foreign language - which they never did - is persuading the viewer that something happened, when it never did.

Whereas, in the AI that I'm proposing, the person literally did move his arms from outstretched to by his side. Except that the sequence is broken by the cut. So there is no fakery involved, merely restoring the sequence that was cut by the edit.

So my proposal involves no or little fakery, since it did happen. Whereas your scenario is total fake, since it never happened.
Offline

ShaheedMalik

  • Posts: 1171
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 11:25 pm

I agree with Marc on this. Switching to a B Cam or C Cam, or B-Roll is the best solution.

Ironically, B-Roll is where AI will make since the most.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 2:35 am

rsf123 wrote:In terms of degree of artificiality, one could say that fabricating the movement of a person's lips to fake them talking in a foreign language - which they never did - is persuading the viewer that something happened, when it never did.

Well, you're hearing the actor speak in a voice and a language that never actually happened in real life... but the difference there is they gave permission for it, they got paid for it, plus it's a film tradition that goes back to the beginning of sound, a hundred years ago. If SAG-AFTRA is for it, then I'm for it. They have said they're find with ADR and dubbing, even if it changes the picture.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 478
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 4:56 am

Marc Wielage wrote: ... Perfect for B-roll ...


Jim Simon wrote: That's when you cut to B-roll. ;)


ShaheedMalik wrote:I agree with Marc on this. Switching to a B Cam or C Cam, or B-Roll is the best solution.

Ironically, B-Roll is where AI will make since the most.


Just to be clear, I am not disagreeing with all your recommendations of using B-roll to cover up cuts. It's a tried and tested method. I would be silly to argue against it, for use in many or most situations.

But what I'm saying is, in my particular videos, I don't think I can benefit from inserting B-roll into cuts. And there will always be other situations where standard techniques are not applicable. In those situations, Ai can come to the rescue.
Offline

Hendrik Proosa

  • Posts: 3238
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 7:37 am

In language dubbing, actors lips have formed all of those positions at one point anyway, so what is fakery and what is not is really a matter of taste. One can say that lip sync is just a bunch of smooth cuts.

As for ”wadddabout the jobs”, no-one in this forum seems to care or bother for the art and skill of roto artists when jumping into magic mask.
I do stuff
Offline

markvansomething

  • Posts: 8
  • Joined: Wed Dec 09, 2020 12:43 pm
  • Real Name: Mark van den bergen

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat Oct 26, 2024 6:30 am

It would be great to have an AI improved version of Smooth Cut.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat Oct 26, 2024 7:52 am

markvansomething wrote:It would be great to have an AI improved version of Smooth Cut.

As was said about 10 messages back, if in frame 1 my arm is stretched over my head and I'm yelling, and in the next frame my hand is in my lap, Smooth Cut would have to morph my hand almost 3 feet. I'll look like an alien from space.

I think people are expecting too much from A.I. It's not going to compensate for bad filmmaking or a lack of editorial coverage. These are basic things you just have to have in order to make the cut work.

BTW, I can and have used Smooth Cut even in features when the editor sped up a scene by cutting a few frames out of the middle, and a coffee cup moved a 1/2". With Smooth Cut, now you can only barely see the movement and it looks flawless. I pointed that out to an editor last year on a show we did, and their reaction was, "ah, I didn't even notice. If I can't tell, then it's perfect!"
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Tom Early

  • Posts: 2711
  • Joined: Wed Jul 17, 2013 11:01 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSun Oct 27, 2024 12:31 am

We should be resisting the ability of AI to distort reality as much as possible.

In the rare event that you cannot somehow replace a jump cut with b-roll or some alternate angle in a way that an audience will actually care about the difference, just suck it up :roll: . Creators have managed without this feature for decades already.

The development of generative AI only increases the power of bad actors to misinform and manipulate public opinion. Your pushing of a major company in video post production to assist in this process is not welcome.
MBP2021 M1 Max 64GB, macOS 15.0, Resolve Studio 19.0.2 build 7
Output: UltraStudio 4K Mini, Desktop Video 12.7
Offline

Rick van den Berg

  • Posts: 1470
  • Joined: Tue Jun 02, 2015 7:47 am
  • Location: Netherlands

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSun Oct 27, 2024 1:56 am

i think we're not far from completely replacing an existing interview, to make it exactly to your liking, or even generating an entire new interview, based on a picture of the person and a picture of the setting. I'm not looking forward to that moment though.
Offline

Ellory Yu

  • Posts: 4316
  • Joined: Wed Jul 30, 2014 5:25 pm

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSun Oct 27, 2024 2:15 am

My want for AI in a product like Resolve is to help cut down the post production effort particularly when sorting out and locating pieces of clips from thousands of files, and placing the pieces in a timeline as I go through editing following a script. If there is an AI feature where you can import a script into a project, and use AI to scan through the media for scenes from the script and line them up in the timeline, even at a 50-60% accuracy, that to me is a 50-60% for that process effort. The current feature, where I have a generated transcript after the fact (medias are already in the timeline) is nice for searching through the clips but I don’t need to use it because by the time I am done with the timeline, I pretty much know where things are and can jump to them without effort… although I am not saying it doesn’t have any usefulness, especially like in cutting interviews or such.
URSA Mini Pro 4.6K G2, Blackmagic Design Pocket Cinema Camera 6K.
PC i7 Workstation Win10 Pro, iMac Pro 27” 5K Retina, iMac i9 27” 5K Retina, MacOS Sonoma
BMD DaVinci Resolve 18.6.6 Studio. BMD Panel and Speed Editor.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSun Oct 27, 2024 3:44 am

Ellory Yu wrote:My want for AI in a product like Resolve is to help cut down the post production effort particularly when sorting out and locating pieces of clips from thousands of files, and placing the pieces in a timeline as I go through editing following a script. If there is an AI feature where you can import a script into a project, and use AI to scan through the media for scenes from the script and line them up in the timeline, even at a 50-60% accuracy, that to me is a 50-60% for that process effort.

We've had this conversation before. The problem with that is -- at least in the films I've worked on -- the actors don't always say the words the way they're presented in the script. There's all kinds of ad-libs and extemporaneous conversation that merely uses the script as a jumping off point. How are you going to find the shots that way? What if an actor reads the same line 10 different ways -- angrily, humorously, sarcastically, dramatically, threateningly, or laughing -- how would A.I. differentiate between them? Sometimes the thin line of emotion and interpretation makes one take vastly different from another, even when the exact same words are being said. [The Oscar-winning editor Walter Murch's famous book, In the Blink of an Eye, goes into this in great detail.]

To me, we've been doing this for almost a hundred years in Hollywood, and the method simply uses an assistant editor (or a team of assistant editors) to prepared lined scripts, create bins, organize timelines, organize scenes, and do all the prep work for the editor. You're basically saying, "how about we use A.I. to do all that?" I say: hire more humans. It doesn't necessarily have to cost an arm and a leg: there are affordable assistant editors out there, especially for small projects and low budget indie films.

Also, an integral part of the editor's job is to watch all the footage, sometimes multiple times, and decide where the best performances are located. Many times, a shot in a film actually comes from a completely different, unrelated scene... but it's a moment that gives the sequence the emotion it needed. I remember keenly some surprising reactions from a certain actress in a film I worked on about 15 years ago, and I asked the editor about it. He told me, "oh, that happened because they were waiting to slate the scene, and you included it in the dailies. The actress had a funny, annoyed look on her face, and somehow it worked for that scene." So it's a moment not in the script at all... it's a surprise moment that happened spontaneously, without planning. A.I. ain't gonna find that. You need a creative human being with an eye towards interesting shots and solid emotion to find things like this.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Ellory Yu

  • Posts: 4316
  • Joined: Wed Jul 30, 2014 5:25 pm

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSun Oct 27, 2024 4:23 am

Marc Wielage wrote:We've had this conversation before. The problem with that is -- at least in the films I've worked on -- the actors don't always say the words the way they're presented in the script. There's all kinds of ad-libs and extemporaneous conversation that merely uses the script as a jumping off point. How are you going to find the shots that way? What if an actor reads the same line 10 different ways -- angrily, humorously, sarcastically, dramatically, threateningly, or laughing -- how would A.I. differentiate between them? Sometimes the thin line of emotion and interpretation makes one take …

Yes, we had this conversation before. I think there are better LLMs that can do predictive analysis on the script and be able to find patterns in the clips that can be deduced upon and analyzed in to be predictably close, not necessarily accurate, and add the part of the clip to the timeline. I’ve used an LLM and RAG application (checkout https://www.katalist.ai/#features ) that took in a script and generated a storyboard that 90% created all the images with angles and perspective that is amazing, there was no second thought from the cast and crew. What I’m saying is that things like this is now common, illustrative, and available, that generative AI can be use in the case for adding the feature we are talking about in Resolve. The thing about having this feature in Resolve is to increase productivity, even with hired assistants in the editing bay. Is adding this feature going to increase the price of Resolve? I don’t know. But even if it doubled, it’s far more cheaper than hiring an assistant editor and training them, etc. And like before, BMD has added so many features to Resolve while still keeping the price unchanged, far more what this ask is. So I guess BMD will still hold on to $295 for a studio license. Anyway, if it can be done, why the resistance? AI doesn’t replace humans, it augments our capacity to improve and be better. After all, Resolve still needs an operator invoke it. If it makes me work faster, smarter, cheaper (more productivity in less time), and NOT harder, then automation, and AI, is a friend of mine.

Sorry that I digress from AI smoothing jump cuts. Let’s get back to OP’s topic.
URSA Mini Pro 4.6K G2, Blackmagic Design Pocket Cinema Camera 6K.
PC i7 Workstation Win10 Pro, iMac Pro 27” 5K Retina, iMac i9 27” 5K Retina, MacOS Sonoma
BMD DaVinci Resolve 18.6.6 Studio. BMD Panel and Speed Editor.
Offline
User avatar

Marc Wielage

  • Posts: 12236
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostMon Oct 28, 2024 12:40 am

Ellory Yu wrote:Sorry that I digress from AI smoothing jump cuts. Let’s get back to OP’s topic.

I just had lunch with a very experienced editor/friend of mine this afternoon, and I mentioned this conversation. Before I could finish, he interrupted and said, "well, that's the assistant editor's job -- let them work that out." So he completely saw my side of it.

As to AI to smooth out jump cuts, there is a point where the morph is just going to look bizarre. There are only so many "fake interpolated frames" the neural engine can create before it starts looking like psychedelic animation. I don't think that's a usable solution.

You could try to use A.I. to create a better wide shot and (say) remove people or add people to the frame, just to give you more interview footage. But... that would mean you've "falsified" footage for the interview. I know the network news organizations would frown on that. An indie film... anything is possible.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline
User avatar

Robert Niessner

  • Posts: 5332
  • Joined: Thu Feb 21, 2013 9:51 am
  • Location: Graz, Austria

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostMon Oct 28, 2024 3:37 pm

Marc Wielage wrote:I just had lunch with a very experienced editor/friend of mine this afternoon, and I mentioned this conversation. Before I could finish, he interrupted and said, "well, that's the assistant editor's job -- let them work that out." So he completely saw my side of it.


I'll agree that for a fictional movie or a doc you will need to skim through all the footage for the gold.
But - please Marc - look beyond your Hollywood bubble - in many other countries around the world an assistant editor is not an option or just not within a budget.
Saying "Thx for help!" is not a crime.
--------------------------------
Robert Niessner
LAUFBILDkommission
Graz / Austria
--------------------------------
Blackmagic Camera Blog (German):
http://laufbildkommission.wordpress.com

Read the blog in English via Google Translate:
http://tinyurl.com/pjf6a3m

Return to DaVinci Resolve Feature Requests

Who is online

Users browsing this forum: No registered users and 14 guests