Jump to: Board index » General » Fusion

Reestablishing Tracking After Blocked by an Object

Learn about 3D compositing, animation, broadcast design and VFX workflows.
  • Author
  • Message
Offline

Silverback

  • Posts: 3
  • Joined: Sat Aug 06, 2022 9:38 pm
  • Real Name: Mark Pikas

Reestablishing Tracking After Blocked by an Object

PostSat Aug 06, 2022 10:04 pm

I've been experimenting with using planar tracking in fusion to experiment with the locked on stabilization effect. I've had some luck, but my footage has the object I'm tracking (a bright red car) going behind trees 2x for a bunch of frames, preventing me from tracking it through the whole clip.

I've tried a number of things, but generally once the object is completely blocked the tracking just stops. A few tries it started tracking seemingly random objects in the frame (nothing else is red). I've tried a bunch of searching and found very little information outside 2 threads mentioning restarting the tracking using the "append" button which I can't seem to find anything like. I've also tried restarting the tracking once the object re-appears from behind the tree and I get an error to the effect of "can't restart without tracking information in the previous frame."

I guess I don't even know if this means reestablishing tracking, doing the tracking differently so it ignores the frames where it goes behind trees, or tracking multiple times on the same clip (and how to do that since from the prompts it appears that it only likes having one set of tracking information in a clip).

Please be gentle, I'm about as novice as they get with this. Details will be helpful.
Offline

xunile

  • Posts: 2653
  • Joined: Mon Apr 23, 2018 5:21 am
  • Real Name: Eric Eisenmann

Re: Reestablishing Tracking After Blocked by an Object

PostSun Aug 07, 2022 2:09 pm

Have you checked out this video?

Win 10 Home | Intel i7 - 10700f 64 GB 1 TB GB SSD 2 TB SSD
RTX-3060 12 GB | Resolve Studio 18.01 | Fusion Studio 18.01

Win 10 Home | Intel Core I7-7700HQ 32 GB 1 TB NVME SSD 1 TB SATA SSD
GTX-1060-6GB | Resolve 17.4.6
Offline

Silverback

  • Posts: 3
  • Joined: Sat Aug 06, 2022 9:38 pm
  • Real Name: Mark Pikas

Re: Reestablishing Tracking After Blocked by an Object

PostSun Aug 07, 2022 11:38 pm

Wow, no I haven't seen that.

It's also way more complicated than I was expecting and doesn't get into picking up the tracking after the object "comes back." I think it might work for me, at least it gives me ideas on how to work around the problems I'm having, but in the end I suspect that someone that knows what they're doing would look at what I'd come up with from this and wonder why I did it the most difficult way possible.

I'm just trying to keep a red blob centered in the middle of the frame, I'm not trying to replace it with something else accurately. I will try it like that, it will definitely teach me something.
Offline

Sander de Regt

  • Posts: 2740
  • Joined: Thu Nov 13, 2014 10:09 pm

Re: Reestablishing Tracking After Blocked by an Object

PostMon Aug 08, 2022 8:02 am

Without actually seeing what you are/were doing it's really difficult to help you.
For example you talk about the planar tracker, but also about the append button. That button is part of the 'regular' tracker. If you're not actually replacing the red blob, there's a good chance that you don't need the planar tracker to begin with. As the name already mentions a Planar tracker wants to track a plane, so it needs info as to what a plane is in your image world. A red blob has no features, so there's no plane to track. A regular tracker will be able to track the blob and then you can use the append function and use this info to center the thing you're following.

Can you share the footage and setup with us, so we can see what you're doing?
Sander de Regt

ShadowMaker SdR
The Netherlands
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostMon Aug 08, 2022 8:08 am

I think it falls into feature request area for Planar Tracker, to actually allow access to its planar surface corner point curves for manipulation. Because then you could do track in two parts and just copy the curves from one tracker to another etc, manually smoothing the range where object is obscured. I tried probing into whether one can actually modify the actual corner point data of planar tracker, but with no success in this thread: viewtopic.php?f=22&t=161261&hilit=+planar
I do stuff.
Offline

Silverback

  • Posts: 3
  • Joined: Sat Aug 06, 2022 9:38 pm
  • Real Name: Mark Pikas

Re: Reestablishing Tracking After Blocked by an Object

PostMon Aug 08, 2022 9:50 pm

Sander de Regt wrote:Without actually seeing what you are/were doing it's really difficult to help you.
For example you talk about the planar tracker, but also about the append button. That button is part of the 'regular' tracker. If you're not actually replacing the red blob, there's a good chance that you don't need the planar tracker to begin with. As the name already mentions a Planar tracker wants to track a plane, so it needs info as to what a plane is in your image world. A red blob has no features, so there's no plane to track. A regular tracker will be able to track the blob and then you can use the append function and use this info to center the thing you're following.

Can you share the footage and setup with us, so we can see what you're doing?


I don't know if this will help, this is a screenshot from the middle of my clip, shot in 4K:
Screenshot 2022-08-08 173628.png
Screenshot 2022-08-08 173628.png (964.77 KiB) Viewed 978 times

The red Cadillac is the object that I want to use as a target for the locked on stabilization. You are in a vehicle driving from right to left handheld camera out of the sunroof, so the clip starts with the Cadillac in front of (to the right of) the tree to the right of it in the foreground of this shot, it goes behind it, comes back out in the frames around this screenshot, then goes behind the clump of trees in the left foreground of this screenshot and comes back out. Most of the clip is before and after these 2 blockages by the trees. I figured the side of the car is my plane, and the clip has a bit of a perspective change, it starts with the camera about 35* behind it, pointing at the back driver's bumper (you can see some of the trunk) and ends about 35* in front of it (you can see headlights and the front of the grill)

The clip is nothing special, it's just a cool car, and "I wonder if I can make this work?"
Offline

xunile

  • Posts: 2653
  • Joined: Mon Apr 23, 2018 5:21 am
  • Real Name: Eric Eisenmann

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 12:46 am

So the Cadillac is stationary and the camera was moving and you want to stabilise the shot so it isn't shaky?
Win 10 Home | Intel i7 - 10700f 64 GB 1 TB GB SSD 2 TB SSD
RTX-3060 12 GB | Resolve Studio 18.01 | Fusion Studio 18.01

Win 10 Home | Intel Core I7-7700HQ 32 GB 1 TB NVME SSD 1 TB SATA SSD
GTX-1060-6GB | Resolve 17.4.6
Offline

Sander de Regt

  • Posts: 2740
  • Joined: Thu Nov 13, 2014 10:09 pm

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 7:47 am

If you want to you can DM me with a link to the whole shot and what you've got so far and I can take a look at it. I'm curious now. Every shot needs its own approach. Did you turn off all the perspective features in the planar tracker? For a shot like this translation and rotation is probably all the stabilization you need or can expect from this distance.
Sander de Regt

ShadowMaker SdR
The Netherlands
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 8:31 am

Everyone seems to get distracted by the example shot, but what about the original question itself: how do you actually stitch together two parts of planar track? Lets just assume for a sec that this is what a specific shot needs, for the sake of technical discussion. There are multiple different ways how I could do it in Nuke for example, but how do Fu experts do it in Fusion?
I do stuff.
Offline

Sander de Regt

  • Posts: 2740
  • Joined: Thu Nov 13, 2014 10:09 pm

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 8:36 am

I wouldn't call myself an expert but in the past I've just 'stacked' various tracks on top of each other. Tracker 1 from 0-50 tracker 2 from 50-100. One tracker's reference at the end, the other one at the start and that's how they ideally blend nicely together.
Sander de Regt

ShadowMaker SdR
The Netherlands
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 12:17 pm

But if you can only track 0-50 and 100-150, then how do you fill the intermediate occluded range? If you can stack one start to others end there is no real full occlusion going on.
I do stuff.
Offline

Sander de Regt

  • Posts: 2740
  • Joined: Thu Nov 13, 2014 10:09 pm

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 1:39 pm

If the thing you're tracking is truly invisible at that point, then it's less relevant and then you can always the track the thing that's occluding it to tie you over.
Sander de Regt

ShadowMaker SdR
The Netherlands
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 3:46 pm

Sander de Regt wrote:If the thing you're tracking is truly invisible at that point, then it's less relevant and then you can always the track the thing that's occluding it to tie you over.

That’s not answering the question though. And if occluding object is on different depth or just moves differently, tracking that is useless.

Partial occlusion starts usually sooner than object becomes fully occluded and this creates a range where track is still needed but trackable area gets smaller up to the point where it can’t be tracked using planar tracker anymore, but object hasn’t fully disappeared yet either.

So, original question still stands: how does one technically stitch together two planar tracks when they do not piggyback and intermediate area should also be planar track covered, so that it all forms one long planar surface movement?
I do stuff.
Offline

Sander de Regt

  • Posts: 2740
  • Joined: Thu Nov 13, 2014 10:09 pm

Re: Reestablishing Tracking After Blocked by an Object

PostThu Aug 11, 2022 4:10 pm

I'm sorry. I thought I did answer the question. Truth is: I don't know how to explain it without using it on a shot. I'm not quite capable enough to put in writing without doing it first.

That's why I was trying to get hold of the footage, so that I could try it and then explain how I did t.
Sander de Regt

ShadowMaker SdR
The Netherlands
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostFri Aug 12, 2022 2:28 am

It can be done. Set your render time from the reference frame to the frame where the occlusion begins. Track those frames. Select your tracked polygon points, drag them on to the occlusion or better on to a surface that has similar movement and reset the end of your render range to the frame where the object reappears. Track through those frames. Select your tracked polygon points, drag them on to the object, adjusting the shape if necessary and reset the end of your render range to the last frame that you need tracked. Track to the end. Then you could go into the spline editor and delete the occluded keyframes.

As for extracting the corner points, I guess that could be done using the corner pin offsets but it would be very laborious. Maybe by, after tracking, lining up and key framing each corner point offset to the relevant corner of the frame on the reference frame and realigning on each frame. That could get you data, whether that's what you want or how you could use it or export it (pen and paper), I don't know. Why you would want to do such a thing?
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostMon Aug 15, 2022 9:01 am

Thanks, tracking something-something, deleting the intermediates etc is a working solution for interpolation in areas of occlusion.

Now, for why one would want the actual corner data? Because then you can, for example:
- Fix problems with each corner manually if necessary. Can't do that with planar track keyframes because they compose all corners and can't properly do it with offsets either for a simple reason: offset needs meticulous changing per each frame to compensate jitter while smoothing can be done by removing keyframes or smoothing the curves themselves.
- Replace some corner data with other data, for example track one corner with regular tracker, doing a hybrid of methods, still resulting in a planar transformation.
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostTue Aug 16, 2022 9:50 am

I can't tell whether you have misunderstood me or not. You can already correct or compensate for a slipping track if you choose Corner Pin in the Operation Mode. If you open up the tab on Corner Pin 1 and 'Reference Time Positions', you will see four offsets that can be keyframed. If you have a four sided plane you can use these to correct a slipping track. You could also I think in some cases, use these offsets to correct none square inserts, providing you you have four coplaner points to use and the insert fits inside the area encompassed by the four points.
You could also use these offsets to record how much each corner of the full rastor has moved, from the reference frame position of x0y1, x1y1 etc. When the track causes perspective distortions the offsets can be used to return the corners back to the reference frame positions. The offsets then represent the difference between the corner positions of the frame, before and after the perspective distortions caused by the track.
If blackmagic could find a way to move the offsets to the track tab it would be ideal, as presently there is no handy way to correct for the other Operation Modes. There may be a way involving inverting the reference time position offsets but you would need to be pretty desperate to go to the trouble.
The adjustment tab in Mocha does something the same as the Reference Time Positions by, I believe, adding another layer but they have done something clever to make it affect the steady operation as well.

edit, There are also four path modifiers in the modifiers tab, these may well allow the track to be adjusted, I haven't gone there yet but it's worth a look.
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostTue Aug 16, 2022 10:14 am

Offset is not an override (unless I’m missing something) so they need to be animated per frame. This is very different from manipulating the original corner position curve. Imagine a value sequence 1.0, 1.3, 1.2, 1.4, 1.1, 1.5 and you want to smooth it to linear curve from 1.0 to 1.5. With offset each of the intermediates must be compensated. With direct curve, just delete the intermediates and interpolate with suitable method. If there is a way to do this for each corner position separately, I’d love to know.
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostTue Aug 16, 2022 11:59 am

You're definitely missing something, they don't need to be animated at all. You can set each of the corners initially and they will simply follow the track. If you set two or more keyframes on any corner, the difference in position between the keyframes is added to the track position for that corner.
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostTue Aug 16, 2022 12:10 pm

I'm talking about fixing the positions of corners. Doing that with offsets needs different offset per each frame if original position is jittering.

So, just to be clear: how can you access the actual position curve for x and y coords per one specific corner point after track has been done? Final screen space position of each corner when planar transformation is applied? I can only find how to access the compound keyframe of whole planar transform per frame, but not decomposed corners themselves.
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 10:05 am

The controls for positioning the each corner of the corner pin are right there when the Corner Pin tab is selected. The movement between frames of each corner of the corner pin, for each frame, is based on the sum of the tracker position movement and the change in the Reference Time Position. Change either and the position of the corner point changes.
Tracker move + RTP move = Point move.
So to get the change in any corner point position due to the tracker with regard to the reference frame position, I can make the point move 0 by moving the corner point back to where it was on the reference frame. The value in the RTP offset must equal the inverse of the Tracker move value with regard to the reference frame..
Here is a simple example in case that explanation is not great.

https://drive.google.com/file/d/1L_jjJ7 ... sp=sharing

Corner Pin 1 is just a simple example, in Corner Pin 2 I have set the corner points in the reference frame to the corners of the rastor. At frames 325,350 and 400 I have repositioned the corners back to the corner of the rastor. The numbers in the offsets at these frames, represent the difference in position, from the reference frame, of all four points, at each of these frames. So to take frame 325 as an example, bottom left began at x0y0 at the reference frame, in order to return that point to the corner of the rastor, I had to offset it x 0.012705 y 0.127814. So the track has moved the corner x -0.012705 y -0.127814 in relation to the reference frame. That's the only way I can see of getting the data you want but it's not very practical.
You will also see in the spline graph that the offsets can be manipulated to some extent both the handles and from side to side in time. This will give you the control of the corner pin you want but as I mentioned, ideally the offsets would be on the track tab. Maybe that is difficult to do or maybe in future.
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 12:18 pm

Thanks for taking the time! What I gather from this is that it is not possible to access the position graphs directly unless this compound you laid out is baked to some secondary curve. This in turn could then drive a cornerpositioner.
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 2:11 pm

Quite the reverse, you don't even need to track a surface, you can position the corners of the corner pin by using just the four point controls and the spline editor if you want. In that case the corners would be driven by only by keyframes with no track data added.
Can you give a practical example of what you want to do with the data. What cornerpositioner are you refering to and what are you trying to do with it?
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 3:17 pm

I know. Context for everything here is fixing and manipulating the result of planar track, specifically accessing the position curves of planar surface corners directly. Hand-animating a cornerpositioner is not related to planar tracking, neither is animating the offsets giving the position data of corners, as was already established. Not sure I can be any more specific, I have described it the best I can in multiple ways, but I’ll try once again.

Planar tracker produces a transformation that expresses a planar transform. This is driving the four corners of established planar surface in screen space. These corners move along a specific path in screen space which is formed from their positions in screen space on reference frame and planar transform matrix that moves them from reference position to position in current frame. Question is very simple: how can one access/generate/modify the final screen space position curves PER corner, per x and y component?
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 3:57 pm

I have already described how the position of the four corner points of a corner pin does not depend only on the the track data but on the sum of the track data and Reference Time Position Data. Therefore for a corner pin to be successful you do not need access to the tracking data but I did describe how you could ubtain it. I'm now very curious, what do you intend to do with that data, since a corner pin can be corrected using the Time Position Data and what 'cornerpositioner' is it you want to affect?
Last edited by birdseye on Wed Aug 17, 2022 4:07 pm, edited 1 time in total.
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 4:07 pm

As I’ve written above, having corner point curves accessible directly is useful for multiple different things. If it were possible directly, there would be no need for any corner positioner. But if it isn’t, one way to get it would be to bake these curves to cornerpositioner corners.
I do stuff.
Offline

birdseye

  • Posts: 182
  • Joined: Fri Jun 12, 2020 2:36 pm
  • Real Name: Iain Fisher

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 4:17 pm

'As I’ve written above, having corner point curves accessible directly is useful for multiple different things'
I'm still curious as to what these multiple different things are in relation to the planar tracker, if i'm missing something, I'm always eager to learn.
If you really need this data for whatever reason wouldn't the best way to get it be to track four coplanar points with a point tracker?
Offline

Hendrik Proosa

  • Posts: 2402
  • Joined: Wed Aug 22, 2012 6:53 am
  • Location: Estonia

Re: Reestablishing Tracking After Blocked by an Object

PostWed Aug 17, 2022 5:19 pm

And why would one need the planar tracker at all if one could just track four points all the time? But just to be complete in the reasoning, it is because planar tracker has a qualitatively different approach to tracking, which gives better results when these four corners can’t be established well enough BUT result still needs full planar transformation.
I do stuff.

Return to Fusion

Who is online

Users browsing this forum: Bryan Ray and 24 guests