Sync Wedding footage

  • Author
  • Message
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Sync Wedding footage

PostSat Aug 15, 2020 7:06 am

Dear BMD,
You have very advanced programming capabilities. Why can't you add just one feature for wedding shooters?
We shoot videos on 3-4 cameras with built-in or ext mic, at different locations of the same venue. These 3-4 cameras have audio.
Plus drone footage without audio.
Syncing all these cameras is critical for all wedding video editors.
Pls, create a feature to accurately sync these.
There can be a way that we manually sync 1 clip of all cameras and Resolve's neural engine takes note of time and audio and then syncs rest of the footage. This is huge. Even plural eyes do not do this. It may be very easy for you. TIA.
PS.: We do not have separate audio-only to sync to. Because all cameras are recording different activity.
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline

Jim Simon

  • Posts: 29796
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Sync Wedding footage

PostSat Aug 15, 2020 3:46 pm

Timecode is the only way this would be possible, and syncing by timecode already exists.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Re: Sync Wedding footage

PostSat Aug 15, 2020 6:44 pm

Jim Simon wrote:Timecode is the only way this would be possible, and syncing by timecode already exists.

I think that is possible only when cameras are time synced before the shoot. What if they are not time synced?
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline

Jim Simon

  • Posts: 29796
  • Joined: Fri Dec 23, 2016 1:47 am

Re: Sync Wedding footage

PostSun Aug 16, 2020 5:03 pm

There are only two ways to automatically sync clips, timecode and audio. If your clips don't have matching audio, then yes, you would need to jam-sync the cameras before hand.

There is no 'feature' BMD could add to overcome that.
My Biases:

You NEED training.
You NEED a desktop.
You NEED a calibrated (non-computer) display.
Offline

Mel Matsuoka

  • Posts: 1179
  • Joined: Wed Aug 22, 2012 9:54 am
  • Location: Clarence, NY

Sync Wedding footage

PostSun Aug 16, 2020 9:54 pm

I truly don’t understand this feature request.

Are you proposing that there should be some way to automatically sync camera footage that doesn’t have matching timecode references OR in-camera audio references?




Sent from my iPhone using Tapatalk Pro
Resolve Studio 18.0.2 / Decklink Mini Monitor / 14" 2021 Macbook Pro Max (macOS 12.5.1, M1 Max) / 32GB RAM
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Re: Sync Wedding footage

PostMon Aug 17, 2020 11:01 am

Jim Simon wrote:There are only two ways to automatically sync clips, timecode and audio. If your clips don't have matching audio, then yes, you would need to jam-sync the cameras before hand.

There is no 'feature' BMD could add to overcome that.





Mel Matsuoka wrote:I truly don’t understand this feature request.

Are you proposing that there should be some way to automatically sync camera footage that doesn’t have matching timecode references OR in-camera audio references?



Sent from my iPhone using Tapatalk Pro


Suppose we have 3-4 cameras each with 250+ clips shot between 6 pm & 7 AM next day and a drone used for 3-4 hrs. How will sync the footage?
Cameras were not synced prior to shoot.
Audio may differ between the cameras and few clips may have overlapping audio
My requirement is to place the clips from every camera in sequence from 6 pm to 7 AM
The clips with overlapping audio should be placed at the same time on different tracks one above the other.
Currently, we place the main camera on the timeline then we search related clips from other cameras and drone to show different angles.
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline

Mel Matsuoka

  • Posts: 1179
  • Joined: Wed Aug 22, 2012 9:54 am
  • Location: Clarence, NY

Sync Wedding footage

PostMon Aug 17, 2020 5:51 pm

I don’t mean to discount the desire to sync cameras under the scenario you describe, but a solution to your problem already exists, in the form of timecode, specifically jam-sync timecode. It’s one of the reasons why timecode was invented, decades ago.

If all of your cameras are recording the same timecode—usually Time-of-Day timecode, for simplicity sake—then they can be synced pretty much instantly in post. Solutions like Tentacle Sync already exist to allow you to do this. It’s a solved problem, and I really don’t know what it is that you expect BMD to even theoretically do to solve this problem in the event that a multicam shoot is not properly shot with synced timecode?
Resolve Studio 18.0.2 / Decklink Mini Monitor / 14" 2021 Macbook Pro Max (macOS 12.5.1, M1 Max) / 32GB RAM
Offline

franciscovaldez

  • Posts: 380
  • Joined: Wed Aug 22, 2012 4:52 pm

Re: Sync Wedding footage

PostMon Aug 17, 2020 11:47 pm

I would try one of the iPhone or android slate apps.

If you have a free running timecode running on the slate, you just need to point your camera to it and then line that number (time) to your timeline.

I'm assuming the slate apps offer this timecode option.
MacBook Pro 13"
M2
UltraStudio 4K

Mac Pro
2.7 GHz 12-Core Intel Xeon E5
64 GB 1866 MHz DDR3
AMD FirePro D700 6 GB
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Re: Sync Wedding footage

PostTue Aug 18, 2020 1:11 am

Mel Matsuoka wrote:I don’t mean to discount the desire to sync cameras under the scenario you describe, but a solution to your problem already exists, in the form of timecode, specifically jam-sync timecode. It’s one of the reasons why timecode was invented, decades ago.

If all of your cameras are recording the same timecode—usually Time-of-Day timecode, for simplicity sake—then they can be synced pretty much instantly in post. Solutions like Tentacle Sync already exist to allow you to do this. It’s a solved problem, and I really don’t know what it is that you expect BMD to even theoretically do to solve this problem in the event that a multicam shoot is not properly shot with synced timecode?

It is not possible to sync the camera time before the shoot. because I do not own all cameras we usually hire a videographer and drone guy. The most photographers I get footage from for editing are not much educated to time sync cameras and 2ndly its not their problem, its editor's.
I tried to manually do it, it is time-consuming. Because after time syncing I need to place each clip one by one in between other camera clips.

franciscovaldez wrote:I would try one of the iPhone or android slate apps.

If you have a free running timecode running on the slate, you just need to point your camera to it and then line that number (time) to your timeline.

I'm assuming the slate apps offer this timecode option.
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline

Mel Matsuoka

  • Posts: 1179
  • Joined: Wed Aug 22, 2012 9:54 am
  • Location: Clarence, NY

Re: Sync Wedding footage

PostTue Aug 18, 2020 4:00 am

Yogendra Singh wrote:It is not possible to sync the camera time before the shoot. because I do not own all cameras we usually hire a videographer and drone guy. The most photographers I get footage from for editing are not much educated to time sync cameras and 2ndly its not their problem, its editor's.
I tried to manually do it, it is time-consuming. Because after time syncing I need to place each clip one by one in between other camera clips.


That's a bummer. Unfortunately, you will have to learn to live with the drudgery of manually syncing the clips yourself.

There's no magic "neural engine" technology that can do this for you. What you are asking for isn't that much different from asking BMD to "just add" a feature that will take a hard drive full of raw footage, then automatically edit everything together to make sense, both story-wise and aesthetically speaking. It's a cool idea in theory, but it's not something that can "just" be added as a feature request to Resolve, because BMD would basically have to invent a revolutionary new AI system that could even come close to doing this on a rudimentary level, let alone to a level that is actually useful.
Resolve Studio 18.0.2 / Decklink Mini Monitor / 14" 2021 Macbook Pro Max (macOS 12.5.1, M1 Max) / 32GB RAM
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostTue Aug 18, 2020 2:27 pm

I wonder if there would be some way to sync clips by luma and hue variations like it’s done by audio waveform now?
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline

franciscovaldez

  • Posts: 380
  • Joined: Wed Aug 22, 2012 4:52 pm

Re: Sync Wedding footage

PostTue Aug 18, 2020 3:54 pm

Gary Hango wrote:I wonder if there would be some way to sync clips by luma and hue variations like it’s done by audio waveform now?


The problem is that they're shooting different stuff. For example a drone footage of the event won't pair with a CU of the bride.
MacBook Pro 13"
M2
UltraStudio 4K

Mac Pro
2.7 GHz 12-Core Intel Xeon E5
64 GB 1866 MHz DDR3
AMD FirePro D700 6 GB
Offline

franciscovaldez

  • Posts: 380
  • Joined: Wed Aug 22, 2012 4:52 pm

Re: Sync Wedding footage

PostTue Aug 18, 2020 4:05 pm

I still think that if you have an external device with running timecode you could do this.

You just need to at any point in time, point the camera to the device.

So let say for example you have a drone and at any point you record a few frames of the timecode device. You can import that clip to a timeline, find the shot of the device, that let's say at that frame reads 02:20:19:13. You mark the clip at that point and align that portion to that corresponding timeline timecode.

You can then go to the beginning of that drone clip, that let's say lands on the 01:59:59:21 on the timeline, you go to clip attributes and assign that starting timecode to the drone footage.

If you do this, you will not only have all your clips align within the timeline but all will have matching timecode.
MacBook Pro 13"
M2
UltraStudio 4K

Mac Pro
2.7 GHz 12-Core Intel Xeon E5
64 GB 1866 MHz DDR3
AMD FirePro D700 6 GB
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostTue Aug 18, 2020 4:11 pm

There would be limitations, but maybe first scan all of them and determine a “master” clip that represents the most variations (probably a CU). Then progressively scan portions of the other clips looking for similar patterns of variations. I wouldn’t say it’s impossible.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline

Hendrik Proosa

  • Posts: 3015
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Sync Wedding footage

PostTue Aug 18, 2020 5:55 pm

A good test for feasibility of such automation is this: can a person with no knowledge of the actual shoot, what was shot and when, sync the clips. If one can, you probably can find some way to automate it. If can’t, well, good luck. Syncing clips by hue variations sounds pretty much impossible to me because differences between cameras will most probably be bigger than differences between different shots from same cam, how on earth could this be synced... Can you say which clip was shot first from ten similar clips from same cam without any knowledge of time and context?

Some form, any form of time is the best bet here, if all else fails I’d look into file creation dates and stuff like that.
I do stuff.
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostTue Aug 18, 2020 6:43 pm

It wouldn’t be so much looking at similarities of hue, but patterns changing from hue to hue. We can manually sync clips by looking at scenes from multiple angles and watching the motion of objects. We see the motion because there are variations of an objects luminance and color as it moves compared to the luminance and colors around it. Syncing by waveform uses patterns of volume changes within different frequencies, not by comparing the actual volume levels. If Pluraleyes can sync a folder full of clips as long as there is some overlap between them, I don’t see why it couldn’t be done visually in a similar way. It would probably be quite difficult to write a program to do it, but someone, some time, will eventually figure it out.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostTue Aug 18, 2020 6:57 pm

Continuing...

At first, you may have to give it “master” clip that shows the action clearly for the entire duration (A cam). A visual pattern file is created from this with timecode assigned. Then all your B cam and drone footage is compared to the visual pattern file of the master and matches are given synced timecode.

As the algorithm is refined, a “master” clip would no longer be needed. Just throw all the clips in a folder and let “Visuasync” do it’s magic.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline

Hendrik Proosa

  • Posts: 3015
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Sync Wedding footage

PostTue Aug 18, 2020 8:00 pm

Pluraleyes can sync because clips contain the same sound. Try syncing clips with totally different sounds/dialogue. Different camera angles are essentially just that. Human vision system makes it look easy, someone moves, just search that “someone” from other clip and sync the move. This isn’t easy by any means for computer vision.
I do stuff.
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostTue Aug 18, 2020 8:05 pm

Never said it would be easy.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Re: Sync Wedding footage

PostWed Aug 19, 2020 2:54 am

Gary Hango wrote:Continuing...

At first, you may have to give it “master” clip that shows the action clearly for the entire duration (A cam). A visual pattern file is created from this with timecode assigned. Then all your B cam and drone footage is compared to the visual pattern file of the master and matches are given synced timecode.

As the algorithm is refined, a “master” clip would no longer be needed. Just throw all the clips in a folder and let “Visuasync” do it’s magic.

would love to have even this.
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline
User avatar

Yogendra Singh

  • Posts: 235
  • Joined: Sun Apr 30, 2017 5:51 pm
  • Location: Chandigarh

Re: Sync Wedding footage

PostWed Aug 19, 2020 2:59 am

Hendrik Proosa wrote:Pluraleyes can sync because clips contain the same sound. Try syncing clips with totally different sounds/dialogue. Different camera angles are essentially just that. Human vision system makes it look easy, someone moves, just search that “someone” from other clip and sync the move. This isn’t easy by any means for computer vision.

PE s just ok, If I throw the whole shoot in it it does not sync properly. I have to group the clips from different cameras at a single spot then I sync and it works fine.
Indian weddings are so noisy that PE often misses syncing.
https://www.instagram.com/weddings_by_yogendra/

Resolve Studio 18.0.1
Fusion Studio 16.2.1
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostWed Aug 19, 2020 1:39 pm

Another option. Make all cameras with an atomic clock sensor built in. That would give every camera in the world the same timecode.

Just thinking out loud.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline

Mel Matsuoka

  • Posts: 1179
  • Joined: Wed Aug 22, 2012 9:54 am
  • Location: Clarence, NY

Sync Wedding footage

PostWed Aug 19, 2020 6:15 pm

Gary Hango wrote:Another option. Make all cameras with an atomic clock sensor built in. That would give every camera in the world the same timecode.

Just thinking out loud.


There was a Kickstarter project aiming to do just this. It appears to have been funded and shipped. I can’t vouch for its real-world efficacy and reliability, though.


https://www.kickstarter.com/projects/ar ... -satellite

When it comes right down to it, while it’s cool to speculate on advanced, AI based solutions to the OPs problem, an actual solution already exists, and has existed for decades now. It seems to me that the OP would be better suited implementing a proper, professional jam-sync timecode based workflow, instead of praying to the BMD gods to develop some kind of heretofore non-existent “neural engine” based intelligent syncing method which probably would never work as effectively as a simple timecode based syncing method would.


Sent from my iPhone using Tapatalk Pro
Resolve Studio 18.0.2 / Decklink Mini Monitor / 14" 2021 Macbook Pro Max (macOS 12.5.1, M1 Max) / 32GB RAM
Offline

Mark Grgurev

  • Posts: 802
  • Joined: Fri Nov 03, 2017 7:22 am

Re: Sync Wedding footage

PostWed Aug 19, 2020 9:44 pm

Hendrik Proosa wrote:Pluraleyes can sync because clips contain the same sound. Try syncing clips with totally different sounds/dialogue. Different camera angles are essentially just that. Human vision system makes it look easy, someone moves, just search that “someone” from other clip and sync the move. This isn’t easy by any means for computer vision.


Gary Hango wrote:Never said it would be easy.


To add to this, there are reasons that audio is used for syncing and not visuals but probably the biggest one is the amount of data you'd have to go through. People have been working with uncompressed video for years because it isn't a lot of data. At 32-bit float, 100 hours of stereo uncompressed audio is 138.24GBs. For comparison, 1080p30 footage encoded with Blackmagic RAW at a 12:1 compression ratio would only record 4.8 hours of footage in that same amount space. That BRAW footage gets decompressed before processing is applied to it though so that's actually 1,658.88 GBs of raw pixel data that it would have to go through for just that one camera.

Syncing things visually like that could take an enormous amount of time and would still be prone to a lot of errors. Just imagine scaling that to more camera and higher resolutions.

Then also consider that, when it comes to syncing things in time, audio has more temporal resolution. We're talking about 48,000 slices of time per-second versus 24 or 30.
Offline

Hendrik Proosa

  • Posts: 3015
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Sync Wedding footage

PostThu Aug 20, 2020 6:18 am

Data rate isn’t the real problem here though. You can convert everything to cga res and it still won’t make heads or tails from it. Audio is synced based on assumption that most of the signal lines up between clips, so it can be compared using frequency patterns as Gary wrote. Images don’t have this property when created from very different angles. You could pretty easily line up clips from same or very similar angle, but aligning a drone shot with MCU will not work based on signal analysis. You need contextual comparison for this. I think we will see something in next five years that tries to do it, but how well it works... we’ll see.
I do stuff.
Offline
User avatar

Gary Hango

  • Posts: 884
  • Joined: Mon Apr 09, 2018 10:35 pm
  • Location: Left Coast
  • Real Name: Gary Hango

Re: Sync Wedding footage

PostThu Aug 20, 2020 1:20 pm

We have code now that can analyze motion in clips breaking it down to motion vectors. A visual syncing program would also look for similar motion vector patterns. It’s all just a matter of number crunching.
Microsoft Windows 10 Pro x64
Intel(R) Core(TM) i7-6700, 3.40GHz, 32.0 GB
MB: MSI, BIOS: American Megatrends Inc. A.60, 12/17/2015
NVIDIA GeForce GTX 960, 2Gb
Resolve 18.1.1.0007(Free)
Offline

Hendrik Proosa

  • Posts: 3015
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Sync Wedding footage

PostThu Aug 20, 2020 2:25 pm

Gary Hango wrote:We have code now that can analyze motion in clips breaking it down to motion vectors. A visual syncing program would also look for similar motion vector patterns. It’s all just a matter of number crunching.

Optical flow analysis for video processing is almost 25 years old, but we still get crappy artifacts from oflow retime. Every problem is just number crunching in the end 8-) I'd put my two cents on auto-grade emerging before visual syncing. Why pay a colorist if you can submit a style guide and push a button...
I do stuff.
Offline

Mark Grgurev

  • Posts: 802
  • Joined: Fri Nov 03, 2017 7:22 am

Re: Sync Wedding footage

PostThu Aug 20, 2020 10:39 pm

Hendrik Proosa wrote:Data rate isn’t the real problem here though. You can convert everything to cga res and it still won’t make heads or tails from it. Audio is synced based on assumption that most of the signal lines up between clips, so it can be compared using frequency patterns as Gary wrote. Images don’t have this property when created from very different angles. You could pretty easily line up clips from same or very similar angle, but aligning a drone shot with MCU will not work based on signal analysis. You need contextual comparison for this. I think we will see something in next five years that tries to do it, but how well it works... we’ll see.


Nah, datarate would definitely be a problem, it just wouldn't be anywhere near the only problem. A computer could theoretically try to matching faces and expression, determine poses, etc. and try to sync from that but programs focused on doing just one of those requires so much computation and still doesn't get reliable enough results. It would all be far too impractical.

The reason I focus on data rate is because it's a simple metric to show how much of an issue this whole task would be.

Return to DaVinci Resolve Feature Requests

Who is online

Users browsing this forum: Bing [Bot] and 15 guests