Align a clip on top of another clip

I have 2 clips of the same scenery. How can I align them, or at least selected parts of them?

If I add 2 tracker points on a window and a door in one clip, and 2 tracker points on the same window and door in the other clip, I want the second clip to be translated, rolled and zoomed so the tracker points align. (I.e., the window and door are kept exactly on top of each other in both clips.)
Is this possible somehow?

(The clips pan 180 degrees. I can easily add trackers, but no tracker point will be visible throughout the whole clip, but maybe that isn't a problem.)

/Marcus

This is probably going to be a relatively difficult track.

I would suggest trying the camera tracker to begin with rather than a planar tracker. While in theory it should be possible to track the footage with the planar tracker by manually changing the tracking points as they begin to pan out of view, I would expect it to be a lot more work that way.

This is probably going to be a relatively difficult track.

I would suggest trying the camera tracker to begin with rather than a planar tracker. While in theory it should be possible to track the footage with the planar tracker by manually changing the tracking points as they begin to pan out of view, I would expect it to be a lot more work that way.

Oh, I don't mind difficulty. I've already done both, actually, but what then? I have my point cloud and camera tracking solved...and then what? How do I match the two? (Or if I do it with manual tracking points, how do I align the two clips?)

/Marcus

Can you show us the clips? Because it's difficult to help you based on just your description.

Can you show us the clips? Because it's difficult to help you based on just your description.

Sure! I have 2 clips, one with a blue excavator and one with a yellow one:

• blue.mp4
• yellow.mp4

I've added them to Fusion, but as you can see from the Merge1 node (with a 0.55 blend) on the right the excavators are not aligned:


Is it possible to track the clips, and translate/roll/zoom MediaInYellow to keep it visually aligned with MediaInBlue?

What is your end goal with the two clips being aligned? What is the final output of the two clips?

What is your end goal with the two clips being aligned? What is the final output of the two clips?

In different clips I have different objects over the same background, and I want to mask in those different objects from the different clips.
Let's say I want to show the excavator from blue.mp4 and the graffiti at the very left from yellow.mp4, and passing cars from both clips.

What is your end goal with the two clips being aligned? What is the final output of the two clips?

In different clips I have different objects over the same background, and I want to mask in those different objects from the different clips.
Let's say I want to show the excavator from blue.mp4 and the graffiti at the very left from yellow.mp4, and passing cars from both clips.

I was just confused when I viewed the two clips since they seem to be clips of the same excavator and the color was changed with a qualifier from the Color tab, everything is identical except a slight difference in the timing of the camera moves. I would think with something like the graffiti you could use a still of that frame and use Match Move and a mask to composite it with the other clip, similar to sky replacement tutorials. You might also be able to use a Paint node in clone mode for the graffiti.

I was just confused when I viewed the two clips since they seem to be clips of the same excavator and the color was changed with a qualifier from the Color tab, everything is identical except a slight difference in the timing of the camera moves. I would think with something like the graffiti you could use a still of that frame and use Match Move and a mask to composite it with the other clip, similar to sky replacement tutorials. You might also be able to use a Paint node in clone mode for the graffiti.

Imagine that instead of graffiti there's a bunch of workers walking around, and also that the blue excavator is moving around inbetween the workers.

Imagine that instead of graffiti there's a bunch of workers walking around, and also that the blue excavator is moving around inbetween the workers.

I would say that if you had a locked down camera and similar lighting conditions, then it wouldn't be that difficult. If you are talking about two different handheld shots, even with the same camera, I would say it would be an order of magnitude harder. When movie studios want to do a handheld shot where they are going to do a composite of multiple takes, they use robotic camera rigs that are able to precisely move the camera the same way each time. They are basically "Faking" a handheld shot so that they can accomplish the composite. When you are talking two real handheld shots you have to deal with different perspectives, parallax, possible differences in focus and a host of other problems. Let's say in one shot you are panning left and the other you are panning right, motion blur would be different in each shot.

I will defer to anyone who has more experience than I do(about 2 years), but I would say that you have set up a real challenge. I wish you luck.

Imagine that instead of graffiti there's a bunch of workers walking around, and also that the blue excavator is moving around inbetween the workers.

I would say that if you had a locked down camera and similar lighting conditions, then it wouldn't be that difficult. If you are talking about two different handheld shots, even with the same camera, I would say it would be an order of magnitude harder. When movie studios want to do a handheld shot where they are going to do a composite of multiple takes, they use robotic camera rigs that are able to precisely move the camera the same way each time. They are basically "Faking" a handheld shot so that they can accomplish the composite. When you are talking two real handheld shots you have to deal with different perspectives, parallax, possible differences in focus and a host of other problems. Let's say in one shot you are panning left and the other you are panning right, motion blur would be different in each shot.

I will defer to anyone who has more experience than I do(about 2 years), but I would say that you have set up a real challenge. I wish you luck.

I understand. However, it wouldn't have to be perfect. As long as I can match a few tracking points on one clip with ones from another clip I'll be fine.

Alternatively, is there some way to stabilize both clips in Fusion? I can't manage to stabilize a clip that has tracking points appearing and disappearing during the clip. It shouldn't be that difficult, right? Like, I have already solved a fairly nice point cloud, so I'd just need a smoothing function for the camera path and make the clip move accordingly.

Stabilization in 2d does not help you here due to image warping that is caused by projection (flat image plane of camera is the cause). Solve your clips as nodal pans, project onto sphere (or use some other suitable method), align cameras in 3d or as a texture in equirectangular space and re-render through other camera to get one match another. Also remove lens distortion, it will throw off your solve and alignment.

If cameras also change position, proper 3d solve and reprojections onto geometry will help align and stick stuff.

Stabilization in 2d does not help you here due to image warping that is caused by projection (flat image plane of camera is the cause). Solve your clips as nodal pans, project onto sphere (or use some other suitable method), align in equirectangular space and re-render through other camera.

Actually I'd be fine with 2D stabilization. It would be close enough for my purpose.
If it's possible to stabilize the 2 clips together, by matching tracking points from one clip to tracking points on the other, then that would be even better, of course.

Or if doing it your way, how would I map the moving clips onto a sphere? (Also, if I did that I wouldn't need a new camera, I could just use the camera from one of the clips, since the other clip is being moved around on the sphere.)

2d stabilization will probably not be close enoguh because you have wide angle lens and massive perspective warp. Same element does not have same size on frame center vs edge. I didn't find a good illustration for this but imagine drawing straight lines (light rays) from sphere surface (world) through the sphere center (centet of lens) to straight line (sensor). Lines that are same angle apart on sphere surface are not same distance apart on straight line, the more angled relative to sensor, the wider the distance, image gets more and mor stretched closer to sensor edges. This is NOT lens distortion, it happens with perfect virtual pinhole camera too. By the way, not taking perspective projection effects into account is the reason why Fu camera projections are warped when mesh is not subdivided enough. The dreaded N word software does not suffer from this and can project accurately onto any low-res mesh.

Projection onto sphere is done with the same solved camera, but depending on what you need to do you might "film" the sphere with some other camera. What the exact node setup is I can never remember by heart because I'm not an everyday Fu user.

2d stabilization will probably not be close enoguh because you have wide angle lens and massive perspective warp.

Sure, but I can time-stretch the top clip to match the rectilinear distortion of the bottom one accurately enough. The seams will mostly be in the middle of the frame, showing the left side of one clip vs the right side of another, like a split screen, and I can choose the seam away from straight lines, so I just want the clips not to be shaking around separately from each other.

In your example shots it appears that your camera also has quite a bit of rolling shutter artefacts when moving, so you'll get in-frame distortion that's different depending on the speed of the pan.
But basically the 'easiest' way to do what you're trying to do, is stabilizing both shots on the same fixed point. (you will be making your work easier if this point is in frame at all times) and then line up those two fixed points and recreate one of both pans after working on those stabilized shots.
It won't be perfect, because there will be quite a lot of wobbeling all over the shots, but it might be enough to suit your needs.

the 'easiest' way to do what you're trying to do, is stabilizing both shots on the same fixed point. (you will be making your work easier if this point is in frame at all times)

Is it possible to stabilize footage in Fusion without using a point in the frame all the time? I have multiple tracking points plopping in and out of existence, but when that happens the scene jumps.

Yes, it is possible. If you use the 2D tracker you have to use the 'append track' option to create steady tracks for the duration of the shot without getting jumps. What is also important is that the tracked points are at approximately the same depth in the scene otherwise your stabilization will be off as well.

I've taken a look at your 2 shots, but even in their most basic motion they show *a lot* of jello and wobbling. I'll see if I can make a screencapture of what I'm seeing shortly. Your footage is severely lacking for what you're trying to achieve.

if there is a lot of rolling shutter distortion, it might not be possible to track these "normally" - you might need to limit the area being tracked to a portion of the clip which is not distorted from the shutter.

In the other software there is a node for that which uses motion estimation to align the shots.