Artificial Intelligence A.I. to smoothen Jump Cuts

  • Author
  • Message
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 12:17 am

For jarring Jump Cuts, as a solution, is it feasible for an NLE software like Davinci to use A.I. to create a blurring effect that is only visible at the frame-level. This blurring effect could be like a morph effect.

What I'm saying is that A.I. could be used to create a blur between the two positions of the person's body, and use a color scheme in that blur that is taken from the color scheme of the person.

I imagine that it would be largely invisible to the human eye because it happens so fast.

But if it were slowed down into slow motion, it would resemble the VFX seen movies where a body transforms into particles.

What I'm suggesting is that A.I. is used to add a color-matched blur between the two positions of the person, at the start and end of the jump-cut, so that when viewed at normal frame rates, the eye does not see a jump cut, but rather a blur that is so fast that it is more visually acceptable than a jump cut.

I realise the "Smooth Cut" already does something like this, but I'm proposing a more radical effect of blurring that can smoothen much more jarring jump cuts. The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.
Offline

Peter Cave

  • Posts: 3913
  • Joined: Thu Aug 23, 2012 6:45 am
  • Location: Melbourne, Australia

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 3:12 am

Have you seen this transition effect anywhere at all?
Resolve 19.0 b3 Mac OSX 14.5 Sonoma
Mac Studio Max 32GB
Offline

sjaehnert

  • Posts: 62
  • Joined: Thu Mar 14, 2019 12:41 am
  • Real Name: Steven Jaehnert

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 3:32 am

two cents: I haven’t been able to use the smooth cut in a way that’s worked for me yet. That said in the interview I tried it on last, the subject was outdoors, moved his face slightly and the person sitting in the background got up and walked behind the subject.


Sent from my iPhone using Tapatalk
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 4:39 am

Peter Cave wrote:Have you seen this transition effect anywhere at all?


No, but I thinking that such an effect could be perceived by the viewer as the person moving from one position to the other - rather than the jump cut that looks like a glitch.

Perhaps the A.I. could review the deleted footage, and mathematically create a transition (from the original footage) to simulate a clean transition, instead of a jump cut.

I thought of these ideas because I'm creating studio videos where the speaker is standing while speaking into the camera, hence, there is more lateral body movement and more hand/arm motion, compared to a seated speaker. Hence, jump cuts don't look good.
Offline

Rick van den Berg

  • Posts: 1424
  • Joined: Tue Jun 02, 2015 7:47 am
  • Location: Netherlands

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue Oct 08, 2019 9:09 am

this might be the most difficult feature request i've ever seen.

also one of those things where i'm like, this should not have any priority at all. but it could be fun.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 14, 2024 5:36 am

I'd like to re-visit this suggestion I made in 2019. Back then, AI hadn't really become to flavor-of-the-month as it is now. Now in 2024 we are seeing OpenAI's Sora creating entire videos.

Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 12:44 am

rsf123 wrote:Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?

Have you tried Smooth Cut? Done in the right way, it can work very well.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Peter Chamberlain

Blackmagic Design

  • Posts: 14116
  • Joined: Wed Aug 22, 2012 7:08 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 1:08 am

rsf123 wrote:For jarring Jump Cuts, ......The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.



try adjusting the smooth cut duration.
DaVinci Resolve Product Manager
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 4:47 am

Marc Wielage wrote:
rsf123 wrote:Hence, it would now seem a small matter for AI to smoothen the gap between edit-cuts of a talking head video?

Have you tried Smooth Cut? Done in the right way, it can work very well.


Peter Chamberlain wrote:
rsf123 wrote:For jarring Jump Cuts, ......The existing "Smooth Cut" seems to work only where the start and end orientations of the person are very similar.

try adjusting the smooth cut duration.



I do use Smooth Cut, but it is no use when the person being filmed waves arms around, such that the hands and arms are in very different positions before and after the cut. That's where I hope A.I. can bridge the gap where Smooth Cut cannot.
Offline

Andy Mees

  • Posts: 3341
  • Joined: Wed Aug 22, 2012 7:48 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 11:31 am

Based only on the premise that such a feature must surely be so very very very far down the list of the things that actually need doing/implementing, and that all those other things must be addressed first, I would gladly give this a hesitant +1. ;)
Offline

Jim Simon

  • Posts: 31211
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 15, 2024 8:52 pm

rsf123 wrote:Smooth Cut, but it is no use when the person being filmed waves arms around,
That's when you cut to B-roll. ;)
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 2:01 am

Jim Simon wrote:That's when you cut to B-roll. ;)

I just fell down laughing because I said the exact same thing at the same time. There's an art to using B-roll footage to bridge difficult edit points.

I'll point out (as I've done before) there was for decades a "BBC Rule" that said if you juxtaposed two different sentences from an interview subject, you couldn't hide the edit and had to just show the jump cut, so that nobody would be accused of obviously trying to re-edit their words to make them say something they never said. For example, if they said, "I am not guilty," and then you eliminated the word "not" and just used Smooth Cut to cover up the gap... now they're saying "I am guilty!" The morality of editing in interviews has to be a consideration.

In the United States, most news organizations just try to use good judgement and never go that far, but they are sometimes guilty of applying a certain "slant" to news stories: like using the takes where somebody is frazzled or awkward, or they're fumbling for the right words. When I was in film school and studied documentary filmmaking, these were hotly-debated topics.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Dermot Shane

  • Posts: 2773
  • Joined: Tue Nov 11, 2014 6:48 pm
  • Location: Vancouver, Canada

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 5:51 am

sjaehnert wrote:two cents: I haven’t been able to use the smooth cut in a way that’s worked for me yet. That said in the interview I tried it on last, the subject was outdoors, moved his face slightly and the person sitting in the background got up and walked behind the subject.


Sent from my iPhone using Tapatalk


i did a similar shot a few years ago, background movement on the B side of the transition made smooth cut unworkable
i roto's the face on both sides of the cut (this was before magic mask)
did a smooth cut on the face only
tracked and comped the new / morphed face on the A side of the shot
was faster than pulling plates and sending to VFX, but sending would have been the better answer had it been flagged eariler
this was in a feature for a studio, the shot passed all QC
Offline

ricardo marty

  • Posts: 1635
  • Joined: Wed Apr 26, 2017 4:03 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 6:16 am

Sometimes you can increase or decrease the size of the shot your cutting to and in certain contents you could put the dip to color effect.

Ricardo Marty
DVR_S 18.5, Asus ProArt PD5, 2.5 GHz i7 16-Core 64GB of 3200 MHz DDR4 RAM GeForce RTX 3070 1TB M.2 NVMe Window 11, LenovoLegion 2.6 i7 10750h 2.6, 64gb 3200mhz, rtx 2060, 1tb ssd M.2 Win 11 BenQ PD3420Q, Sony FS700R, Bmp4k, Sony A6700. PreSonus AudioBox
Offline

ricardo marty

  • Posts: 1635
  • Joined: Wed Apr 26, 2017 4:03 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 6:16 am

Sometimes you can increase or decrease the size of the shot your cutting to and in certain contents you could put the dip to color effect.

Ricardo Marty
DVR_S 18.5, Asus ProArt PD5, 2.5 GHz i7 16-Core 64GB of 3200 MHz DDR4 RAM GeForce RTX 3070 1TB M.2 NVMe Window 11, LenovoLegion 2.6 i7 10750h 2.6, 64gb 3200mhz, rtx 2060, 1tb ssd M.2 Win 11 BenQ PD3420Q, Sony FS700R, Bmp4k, Sony A6700. PreSonus AudioBox
Offline

Steve Fishwick

  • Posts: 1188
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 16, 2024 10:39 am

Jim Simon wrote:That's when you cut to B-roll.


As Jim implies; AI should never replace the good old fashioned c/away. Maybe I'm old fashioned but in over 35 years in the cutting room and increasingly having to deal with 'spray and pray' hand held reality rushes; there has never been a time I had to leave a jump cut in (except for effect in a music sequence perhaps), because there was no other solution; no matter how bad the rushes - there is always a more elegant solution.

When I shoot my own footage; because I am an editor in background, I will shoot interviews on sticks and then ask the question 3 times; each with either a MLS, MS or MCU/CU, so I can cut in and out with impunity, for emphasis; even the best camera person can miss the emotional moment by zooming in too late. Whereas, even when they don't answer exactly the same each time, you can always find the cut in, at the right point.

Obviously with 4K and plus you can just reframe too. Cutting in on the same angle used to be known here as the 'American Cut', I don't know why and many still dislike it, I don't, quite the opposite often, as long as the angle changes sufficiently.
Offline

Jim Simon

  • Posts: 31211
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 1:05 am

Steve Fishwick wrote:there has never been a time I had to leave a jump cut in
Dude, I'm beggin' you...OFFER A COURSE FOR YOUTUBERS! ;)

90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 5:30 am

Jim Simon wrote:
rsf123 wrote:Smooth Cut, but it is no use when the person being filmed waves arms around,
That's when you cut to B-roll. ;)


For my videos, they're like interviews with a person where you don't go jumping to B-roll. Moreover, the cuts are quite frequent, so it'd look funny with numerous jumps to B-roll.

In the era of DALL-E and SORA that can create videos of everything, it might not be too big a hill to climb to just fill in the gaps between a Jump Cut.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 6:38 am

Jim Simon wrote:90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".

Yeah, a lot of that is distracting and bizarre. I get that it's a "style," but it doesn't exactly look real and natural to me.

Two of the things we often did in local TV news: we'd get a reverse angle of the interview subject from the back while the interviewer was nodding and/or taking notes. Perfect for B-roll when you had to (say) cut a minute out of the middle of an interview and juxtapose two sentences together just to get to the point.

And another was the (somewhat amateurish) "shot of the interviewee's hands moving," without seeing their face in the shot. Not exactly smooth, but for some it's better than a jump cut.

The third thing I often see on the big news shows (60 Minutes, 20/20, etc.) is going to the "Killer Wide Shot" from the back of the room that shows the lighting setup, the cameras, the sound guy, the director, and so on. If it's far back enough, you can't see the people's mouths enough to worry about lip sync. So that'd be a worst-case scenario of a 3rd angle to fall back on.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

Steve Fishwick

  • Posts: 1188
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 9:05 am

Jim Simon wrote:Dude, I'm beggin' you...OFFER A COURSE FOR YOUTUBERS!

90% of the clips I'd like to check out are absolutely unwatchable with jump cuts. Even coming from (ahem)..."professionals".


:lol: Thanks Jim but I fear it would be wasted; though I've thought about it and do teach from time in colleges; and I agree with you totally!
Marc Wielage wrote:Yeah, a lot of that is distracting and bizarre. I get that it's a "style," but it doesn't exactly look real and natural to me.

Two of the things we often did in local TV news: we'd get a reverse angle of the interview subject from the back while the interviewer was nodding and/or taking notes. Perfect for B-roll when you had to (say) cut a minute out of the middle of an interview and juxtapose two sentences together just to get to the point.

And another was the (somewhat amateurish) "shot of the interviewee's hands moving," without seeing their face in the shot. Not exactly smooth, but for some it's better than a jump cut.

The third thing I often see on the big news shows (60 Minutes, 20/20, etc.) is going to the "Killer Wide Shot" from the back of the room that shows the lighting setup, the cameras, the sound guy, the director, and so on. If it's far back enough, you can't see the people's mouths enough to worry about lip sync. So that'd be a worst-case scenario of a 3rd angle to fall back on.


Yes Marc, all witnessed and experienced here too - Oh God, the meaningless hand shot they used to take :lol:

If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Most good camera people now will be more creative and purposeful with their B-roll. It could be family photos in the room, even a shot out of the window etc. Often there is a second camera from the side too. Years ago there was a brilliant filmmaker here, I forget his name, who made a film about the Everly brothers; and they being twins he shot Phil straight on and Don side on, and it was a very powerful interview.

If you think about it, and this is what I do teach; editing is the most cinematic of all cinema's elements: Photography, sound, music etc. all come from elsewhere; but the pure juxtaposition of shots, is unique to the moving image. And the reaction shot can be the most cinematic of all.

If there is a conversation between two or more people, it is not always the person talking who is the point of interest; as in life. The other person's reaction might tell the story better.

And the example I always use is from The Hustler, cut by the brilliant Dede Allen; where George C. Scott is haranguing Paul Newman, Fast Eddie, telling him he's finished; Everyone is just looking at the floor embarrassed; and it's all a series of reaction shots.
rsf123 wrote:For my videos, they're like interviews with a person where you don't go jumping to B-roll. Moreover, the cuts are quite frequent, so it'd look funny with numerous jumps to B-roll.

In the era of DALL-E and SORA that can create videos of everything, it might not be too big a hill to climb to just fill in the gaps between a Jump Cut.


My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar. Though I will add this, if it's absolutely necessary (Court legal Video may be an instance) then it is much better to leave it as a jump-cut; since in my mind some kind of attempt to be invisible, is dishonest - you are manipulating what was said rather than acknowledging editorial choice.

One day we'll be able to just sit back like drones and let AI do everything; and have to do nothing creative ever again. I love AI as an aid but I never ever want it to cut for me.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 1:44 pm

Steve Fishwick wrote:[

My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar.


Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.

The videos I make are talking head videos where the subject matter is highly technical, tertiary level, postgraduate subject matter. Hence, during filming, the person often tries several takes to explain the same point. Then, during editing, we find the clip which was the best way of explaining, and then have to splice it together with other parts of the filming to produce what appears to the viewer as a seamless single presentation.

There are no B-roll. There are diagrams inserted into the video, but always with the speaker visible, because the speaker points to the diagrams to explain them.

In this scenario, I try the best to merge portions of videos into a seamless presentation. The hardest parts are where the ends being joined consist of the speaker having vastly different hand gestures.

That is why I thought AI might come to the rescue.
Offline

ShaheedMalik

  • Posts: 884
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 2:57 pm

rsf123 wrote:
Steve Fishwick wrote:[

My friend, I can't ever imagine what kind of interviews would they be; better to 'jump' to something meaningful than jump-cut and jar.


Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.

The videos I make are talking head videos where the subject matter is highly technical, tertiary level, postgraduate subject matter. Hence, during filming, the person often tries several takes to explain the same point. Then, during editing, we find the clip which was the best way of explaining, and then have to splice it together with other parts of the filming to produce what appears to the viewer as a seamless single presentation.

There are no B-roll. There are diagrams inserted into the video, but always with the speaker visible, because the speaker points to the diagrams to explain them.

In this scenario, I try the best to merge portions of videos into a seamless presentation. The hardest parts are where the ends being joined consist of the speaker having vastly different hand gestures.

That is why I thought AI might come to the rescue.


Using a second camera or B-Roll would be still the best solution. Even with those diagrams, you can shoot the diagrams as secondary shot. Then back to shot of the speaking pointing to them.

Take a look at the Multicam Editing tutorial.

https://www.blackmagicdesign.com/produc ... e/training
Offline

Steve Fishwick

  • Posts: 1188
  • Joined: Wed Mar 11, 2015 11:35 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostFri May 17, 2024 3:48 pm

rsf123 wrote:Perhaps rather than using your scenarios as a reference point for evaluating mine ... you could offer some comments on my scenario.


I'm Sorry but you can hardly blame me for that; not knowing your particular example. As Shaheed suggests, you can use the diagram inserts as full frame shots.

If you'll forgive me whilst I may surmise you are extremely knowledgeable in your field, you tend to suggest you are very inexperienced in filmmaking and I would therefore urge you, rather than just dismiss my examples, seek out other examples more relevant, of how such material may be treated and approached, IMV more properly. So when the speaker moves to indicate something on the diagrams, the logic of cutting follows, in that their hand or pointer will be in shot too, for example, and your intended audience, will not be confused or jarred by an unnecessary jump-cut.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 2:59 am

Steve Fishwick wrote:If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Well, there's no harm in shooting a big wide reverse with just a light and a camera in the frame, with the back of the interviewee's body to camera. At least it's coverage done on the day of the interview, even if there's technically nobody else sitting in the chair. Often, it's just a segment producer or somebody else who doesn't necessarily want to be seen on camera -- no biggie.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline
User avatar

Charles Bennett

  • Posts: 6511
  • Joined: Sat Nov 05, 2016 11:55 am
  • Location: United Kingdom

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:16 am

What really annoys me on jump cuts is not so much the picture but the clipping of the front and end of the dialogue removing the natural flow of speech. This is a growing trend amongst YouTubers to the point where some videos are just unwatchable.
Resolve Studio 19.0b3 build 33
Dell XPS 8700 i7-4790, 24GB RAM, 2 x Evo 860 SSDs, GTX1060/6GB (551.86 Studio Driver), Win10 Home (22H2), Speed Editor, Faderport mk1, Eizo ColorEdge CS230 + BenQ GW2270 + Samsung SA200, Canon C100mk2, Zoom H2n.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 2:29 pm

Marc Wielage wrote:
Steve Fishwick wrote:If there is a journalist/interviewer, very sound advice and I would/do shoot the reverse too. Most films I work on now there will be a master interview, where there is no interviewer, i.e the researcher is asking the questions with the subject's eyeline looking off camera at them; and their questions removed in the edit.

Well, there's no harm in shooting a big wide reverse with just a light and a camera in the frame, with the back of the interviewee's body to camera. At least it's coverage done on the day of the interview, even if there's technically nobody else sitting in the chair. Often, it's just a segment producer or somebody else who doesn't necessarily want to be seen on camera -- no biggie.


In these days of pandemic zoom calls, where everything in frame looks immaculate, but all the garbage in the room is shoved out of camera view? Well, it's the same with many smaller studios, particularly home studios. Virtually all the suggestions made in the thread pertain to professional studios. I can't even fit a 2nd camera for angle shots in the room that functions as my studio. But no one needs to know that from the video image with a nice background.
Offline

Jim Simon

  • Posts: 31211
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:03 pm

Charles Bennett wrote:What really annoys me on jump cuts is not so much the picture
That is the most annoying part. It's very visually distracting.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Jim Simon

  • Posts: 31211
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 7:08 pm

rsf123 wrote:I can't even fit a 2nd camera
You don't necessarily need multiple cameras.

You just need ONE GOOD TAKE.

If that requires a dozen tries, do them.

Because to paraphrase Gene Krantz..."Jump Cuts are NOT AN OPTION!" ;)
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostSat May 18, 2024 11:45 pm

Jim Simon wrote:
rsf123 wrote:I can't even fit a 2nd camera
You don't necessarily need multiple cameras. You just need ONE GOOD TAKE. If that requires a dozen tries, do them. Because to paraphrase Gene Krantz..."Jump Cuts are NOT AN OPTION!" ;)

It's not always possible to do multiple takes, particularly on an interview. I'm reminded of a situation on the Red User Forum about 8-9 years ago where somebody was interviewing an Olympic athlete on the verge of a nervous breakdown, and they dissolved into tears on camera while telling a harrowing story of their personal grief that kept them off the Olympic team. And just at that moment, the Red camera overheated... and the fan blew so loud it obscured her sobs.

Granted, documentaries and Zoom calls are kind of a special case, but even there, it's possible to shoot coverage. One thing we tell clients is to shoot a reaction shot of the person watching the zoom call (like an over-the-shoulder), and that might be a good moment to jump to the next answer. A skilled editor can make it work without straining Smooth Cut. And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 12:37 am

Marc Wielage wrote: And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.


Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.
Offline

Videoneth

  • Posts: 1818
  • Joined: Fri Nov 13, 2020 11:03 pm
  • Real Name: Maxwell Allington

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 12:22 pm

"AI" is not "magic". :D

If training a model to specifically make the "smooth cut" transition better-looking, why not?

I use the "smooth cut" very often, when the only things that really move a little bit are the eyes, mouth, things like that. I don't care too much with things jumping out in the background, it is what it is, there is no way around it when it's the same scene.

I don't like the morphing effect but it can be use for artistic purpose (which I did in the past).

But it's such an elaborate process for a tiny, tiny problem that doesn't need to be corrected imo.

I personally don't care about jump cuts; they are totally fine (except when it's to remove silence after every single words). But craming b-rolls everywhere just for the sake of hidding everything is worse imo.

I've seen those, lol. Like, dude, you're making a tutorial; don't try to make something cinematic (or whatever) for this 8-minute video. Adding fluff that adds no value to anything is more annoying than removing something with no value like silence (which can become a jump cut).

I think the OP forgot to think about the audio part. Most jump cuts are there because words or silences were removed. 

Imagine you have someone with their hand up in the air, and the hand is slowly raised before the cut. Then the hand is on the bottom of the frame. How does that work? You add a "smooth-AI - Cut" between the clips, a 2–5 sec. transition, so it would look more natural and not as if the person is slaming their hand.

The "AI" would analyze the two frames and create all the elements in-between, and assuming the result is good, you'll end up with a gap for the audio that needs to be filled.

At the end of the day, audio is 10 times more important than video anyway. People can watch the most compressed, out-of-focus video with very good audio. But they will 100% complain if the images are pixel-perfect with crappy audio.

You can "zone-out" while watching a video with a tons of jump cuts (like the earlier videos of Mr Beast, one example that comes to mind, I couldn't watch them years ago because they "stressed" me out lol), and still have a good experience because the audio is good. The opposite doesn't work well 100% of the time.
Windows 10
19b3
nVidia 3090 - 555.85
Offline

ShaheedMalik

  • Posts: 884
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostTue May 21, 2024 1:37 pm

rsf123 wrote:
Marc Wielage wrote: And there are definite limitations with Smooth Cut, to the point where it makes the human body look like an alien, bending into impossible positions. No amount of smoothing can get a human character from "here" to "there" without some major conniptions.


Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.


You if you have to put a jump cut in crucial explanation, then it is a poor explanation and you have to redo it anyway. You want AI to make up for poor filmmaking.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 3:48 am

ShaheedMalik wrote: You want AI to make up for poor filmmaking.


Actually, yes. In an ideal world, the speaker will roll out pearls of wisdom in their speech with the ultimately-good version being the first and only take. But in the real world, we do make errors, and require many takes/tries/attempts. And sometimes, the good stuff is spread across a few takes, and needs to be stitched together to give an appearance of being seamless.

So, yes, I want A.I. to make up for poor filmmaking.
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 3:54 am

Videoneth wrote: The "AI" would analyze the two frames and create all the elements in-between, and assuming the result is good, you'll end up with a gap for the audio that needs to be filled.


No. Instead, I envisage that A.I. can offer to overwrite the portion of your video, before and after the cut. Hence, there would be no break in the audio.

Example:

Clip 1, the person says, with arms spread wide apart, "I caught a massive fish and it was // ... er, what type of fish, hmm, I forget, I think .."

Clip 2:, the person says, with arms by their side, "and it was // an Atlantic Salmon".

If you cut out the part where he forgot his lines, you end up with a sudden jump where the arms are spread-eagled apart ... suddenly cut to arms being by his side.

So the AI would re-create and REPLACE the video on either side of the cut, so that the AI footage would show the arms coming together. And this movement of the arms can be deduced from the other footage where the arms go from position 1 to position 2.

Hence, by the AI creating and replacing footage either side of the cut, it gives a smooth arm movement with no cut.

No gap in the edited audio, ""I caught a massive fish and it was an Atlantic Salmon."

I have been experimenting with written-prompts that instructs ChatGPT OpenAI how to create diagrams, and I feel that the above is well within the capability of OpenAI's algorithms.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 6:34 am

rsf123 wrote:Hence, you're agreeing that, no matter how many conventional techniques - like B-roll and different angles -there will be situations that those tried-and-trusted techniques just will not work. And that's where A.I. becomes the future tool of trade. It's just that we've never been there before. It's the new frontier.

Well, the B-roll cutawway will always work. If the guy says, "I caught a fish this big," and we cut to a still of an enormous fish and then back to him, the audience will get it. A lot depends on your own imagination and the available stock footage (or still frames or whatever you have).

For example, you do not want to cut to B-roll or rear view, when the person is in the middle of a crucial explanation.

Sure you can, if the audio continues underneath. This is a documentary technique that goes back almost 100 years, literally to the beginning of sound films

Marc, have you seen the videos of what OpenAI's Sora can do? If you have, you'll realise that anything we can image, now is possible. The only limitation is the BMD Developers' time and financial resources to achieve it.

Sure, but I'm kind of morally opposed to the amount of fakery in OpenAI imaging, especially if it's used to bend a human being into saying or doing something he or she didn't actually say or do. There are uses of AI that I'm OK with: for example, if you released a foreign version of a Hollywood film, and you wanted the American actors' lips to move along with the foreign-language dialogue. That's been demonstrated and it can work. Morally speaking, I have no problem with it. I do have a problem with AI if it's stealing pieces of somebody else's creative work, or if it's putting human artists (or actors or technicians) out of work.

I'd point to the recent lawsuit between actress Scarlett Johansson and the Chat-GPT people as an example of AI gone horribly wrong...

Scarlett Johansson told OpenAI not to use her voice
https://www.theverge.com/2024/5/20/2416 ... gal-action
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 9:26 am

Marc Wielage wrote: I'm kind of morally opposed to the amount of fakery in OpenAI imaging, especially if it's used to bend a human being into saying or doing something he or she didn't actually say or do. There are uses of AI that I'm OK with: for example, if you released a foreign version of a Hollywood film, and you wanted the American actors' lips to move along with the foreign-language dialogue. That's been demonstrated and it can work. Morally speaking, I have no problem with it. I do have a problem with AI if it's stealing pieces of somebody else's creative work, or if it's putting human artists (or actors or technicians) out of work.


In terms of degree of artificiality, one could say that fabricating the movement of a person's lips to fake them talking in a foreign language - which they never did - is persuading the viewer that something happened, when it never did.

Whereas, in the AI that I'm proposing, the person literally did move his arms from outstretched to by his side. Except that the sequence is broken by the cut. So there is no fakery involved, merely restoring the sequence that was cut by the edit.

So my proposal involves no or little fakery, since it did happen. Whereas your scenario is total fake, since it never happened.
Offline

ShaheedMalik

  • Posts: 884
  • Joined: Wed Aug 19, 2020 5:28 am
  • Real Name: Shaheed Malik

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostWed May 22, 2024 11:25 pm

I agree with Marc on this. Switching to a B Cam or C Cam, or B-Roll is the best solution.

Ironically, B-Roll is where AI will make since the most.
Offline
User avatar

Marc Wielage

  • Posts: 11284
  • Joined: Fri Oct 18, 2013 2:46 am
  • Location: Hollywood, USA

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 2:35 am

rsf123 wrote:In terms of degree of artificiality, one could say that fabricating the movement of a person's lips to fake them talking in a foreign language - which they never did - is persuading the viewer that something happened, when it never did.

Well, you're hearing the actor speak in a voice and a language that never actually happened in real life... but the difference there is they gave permission for it, they got paid for it, plus it's a film tradition that goes back to the beginning of sound, a hundred years ago. If SAG-AFTRA is for it, then I'm for it. They have said they're find with ADR and dubbing, even if it changes the picture.
Certified DaVinci Resolve Color Trainer • AdvancedColorTraining.com
Offline

rsf123

  • Posts: 468
  • Joined: Tue May 21, 2019 3:05 pm
  • Real Name: Roger Smith

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 4:56 am

Marc Wielage wrote: ... Perfect for B-roll ...


Jim Simon wrote: That's when you cut to B-roll. ;)


ShaheedMalik wrote:I agree with Marc on this. Switching to a B Cam or C Cam, or B-Roll is the best solution.

Ironically, B-Roll is where AI will make since the most.


Just to be clear, I am not disagreeing with all your recommendations of using B-roll to cover up cuts. It's a tried and tested method. I would be silly to argue against it, for use in many or most situations.

But what I'm saying is, in my particular videos, I don't think I can benefit from inserting B-roll into cuts. And there will always be other situations where standard techniques are not applicable. In those situations, Ai can come to the rescue.
Offline

Hendrik Proosa

  • Posts: 3137
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Artificial Intelligence A.I. to smoothen Jump Cuts

PostThu May 23, 2024 7:37 am

In language dubbing, actors lips have formed all of those positions at one point anyway, so what is fakery and what is not is really a matter of taste. One can say that lip sync is just a bunch of smooth cuts.

As for ”wadddabout the jobs”, no-one in this forum seems to care or bother for the art and skill of roto artists when jumping into magic mask.
I do stuff

Return to DaVinci Resolve Feature Requests

Who is online

Users browsing this forum: No registered users and 14 guests