Correct aspect ratio with planar tracker corner pin

I have a question about corner pinning with the planar tracker. Basically I'm trying to figure out how to make sure that the image/footage that you composite into the corner pin is the correct aspect ratio on the surface it's being composited onto. I feel like this issue isn't very complicated at heart, but I am going to give a long explanation of it just to make sure my question is actually clear since last time I tried to explain I didn't do a very good job. (For what it's worth, I don't fault anyone for not having the time or patience to read all of this, but any help would be extremely appreciated.)

First, the image in the corner pin, at least by default, stretches to the four corners of the corner pin. If you have markers in your footage, this is probably not an issue, but if you don't, there are many circumstances where the corners of the corner pin aren't where the corners of the composite are intended to be. For example, it's common to use lines known to be parallel or perpendicular in the real world to define the correct perspective with the corner pin, even though these might be beyond the intended corners for the composited image. As I mentioned, this can easily be worked around by inserting a transform node between the media and the tracker node, which then allows you to adjust size and position relative to the plane defined by the corner pin. However, this illustrates the aspect ratio I'm talking about pretty well: as far as I know, there isn't a way to get the comped image to be the correct aspect ratio without eyeballing it. Furthermore, what really bugs me about it is that I'm pretty this can be computed from the data already defined by the original corner pin points. I'm going to go on a short math rant for a second, but then I'll give a very clear visual example.

CornerQuestion02.png (236.76 KiB) Viewed 2344 times

My first image shows a few examples of the corner pin tool with "show grid" enabled, and while I don't know the proper mathematical proof for it, it's pretty easy to see that any 4 point polygon in screen-space can define a rectangle in world-space relative to the viewing angle. This essentially boils down to the idea that any 4 points in screen-space correspond to only one possible rectangle on a world-space plane. For the sake of further clarification (and/or confusion), I should add that this does not mean only one possible world-space rectangle, but a set of planes that are all parallel, whose orientation is defined by the original polygon, and for each of those planes there is only one possible rectangle that matches the screen-space polygon. All of these possible rectangles would appear the same size in screen-space, and would correspond to larger world- space rectangles for planes farther away, and smaller ones if the plane is closer. This is the geometric consequence that follows from the assumption that all the angles of the screen-space polygon are actually 90 degree angles on a plane that is being seen from a viewpoint that isn't perpendicular to it.

That said, I'm pretty sure that it's a necessary mathematical consequence that all of those possible world-space rectangles are of one specific aspect ratio, regardless of them not necessarily being any specific world-space size. What this means is that as soon as you pick ANY 4 points with the corner pin, you have already defined the only possible aspect ratio for every rectangle that could possibly be seen as that specific shape from that specific viewing angle. And while I don't know how to do this calculation off the top of my head, I feel like it is definitely way less complicated than pretty much all the other math the planar tracker already does. I'm kind of just wondering why there isn't a "maintain foreground aspect ratio" checkbox right under the "show grid" checkbox.

So back to the real question with a more practical example. Let's say there is a space on a plane in your shot that you want to composite something onto, and the footage wasn't shot with markers. That's fine though, because there are lines that you know are parallel and/or perpendicular in world-space. You use them to define the points of the corner pin, as this allows you to get the perspective correct, but those points aren't where you want the corners of the composite to be. Like I mentioned before, this is where I use a transform node to adjust within the corner pin, but I'm only able to do that by eye, and it feels very wrong to have to eyeball something that I know has only one mathematical solution for which I already have the necessary data.

CornerQuestion05-2.png (407.09 KiB) Viewed 2344 times

Here are some images that show that scenario pretty well. In the first image (top left), let's say you're trying to comp an image into the area where I poorly drew the green square. For the sake a sort of extreme example, we're also going to imagine that the corner and line that I covered with red is occluded by something in the shot (if it wasn't, you could accurately corner pin that one metal area, but you would still have the same problem I'm describing). Since the top and bottom edges of the left metal area are clearly the same height as the first one, we can use the left corners from that one instead to still accurately define the plane we want to composite to (top right). Obviously, this will probably get you a super stretched composite (bottom left). In this case, the test image I am using for the composite is originally a square. And finally, the bottom right image shows the way I currently know to adjust the comped aspect ratio, which requires eyeballing it. The other work around I've found is to use the ImagePlane3D node for the transform changes, since you can leave the aspect ratio alone and line it up using the pitch and yaw controls; but this means you have to eyeball getting it on the correct plane, which feels like an even worse compromise than eyeballing the aspect ratio.

Sorry for making this post so long, but hopefully I at least made my question clear even it was at the expense of the length of this post.

If you know the physical size of the thing being tracked, then you can use a Crop node to enlarge your element's raster to fit its aspect.

That's good to know, however I'm really trying to find a solution for cases where I have no information at all. I'll keep that in mind for when I'm on set though.

If you don't have the information about the physical set, then it's down to some photogrammetry techniques. Fspy to estimate lens and camera location, and then some geometry(like, math geometry, not mesh geometry ) to try to calculate relative sizes. Or eyeballing it, which is usually faster and often close enough, depending on what you're doing.

Cody, I'm wondering if you're getting confused regarding the difference between the 'tracking area' and the 'corner pin area'?

A couple of things you've said in this post and others made me think that perhaps you're thinking they're one and the same - that you track an area, then you pin to that same area, and that could lead to problems when you have to track an area much wider than the target?

If so, then that is not the case. They are in fact quite distinct. The only requirement is that they're co-planar - meaning that they share the same planar surface as seen by the camera.

From the Resolve manual:

TIP: Do not confuse the pattern you’re identifying with the region you’re planning to corner pin (which always has four corners and is separately specified in Corner Pin mode.

From the Mocha Pro manual:

One of the most important concepts to understand with the Mocha planar tracking system is that the spline movement is not the tracking data.

It’s best to think of the splines you draw around objects as search areas. Here’s a breakdown of how the tracking works:

1. By default, any spline you draw is linked to the tracking data of the layer it is currently in. In hierarchical terms, the spline is the child of the track, even if there is no tracking data.

2. When you begin to track a layer, the area of detail contained within the spline(s) you have drawn will be searched for in the next frame.

3. If the planar tracker finds the same area in a following frame, it will tell the tracker to move to that point. Because the spline is linked to the track by default, it will also move along with it and the search begins again for the next frame.

This is because the spline is linked to the track, but the track is not linked to the spline. The spline is merely a search area to tell the track where to go next. It is a common misconception that moving the spline while tracking is affecting the movement of the tracking data. It is not. Moving the spline is only telling the tracker to look in a different place and will not directly affect the motion of the tracking.

Therefore, going back to your question: it does not matter if you have to use the whole large rectangular area when tracking. You can still then create a Corner Pin area using any subset of that wall, and I believe you would be recommended to do that rather than trying to corner pin using the same area you tracked and then manipulate the image to look right in that.

So in your wall example, you can define the Corner Pin area using the small square on the far right where you actually want the image to go. You can eyeball those four points, or try to calculate the exact positions using the methods Bryan described.

But that Corner Pin step is separate and distinct from the earlier tracking, the purpose of which is just to find a surface that's co-planar with the target area, and suitable for tracking. This could even be a completely different area to your target - like that lower blue stripe in your wall example (as long as it's co-planar with regards to the camera movement, ie not raised away from the rest of the wall or something.)

Here's a screenshot from a Mocha tutorial. In this example, the tracking area is the red square at the top with a white dotted line around it - this is the equivalent of the polygon you draw in the Fusion Planar Tracker's Track mode. It's tracking only the top edge of the phone, due to the screen having too many reflections on it.

Then the planar surface - the equivalent of the Fusion Corner Pin Area - is the blue rectangle in the middle with a + in the middle (he's just about to drag it down to cover the whole phone):


(later in the tutorial he adds another tracking area for the bottom edge as well, providing two tracking areas to Mocha, both of which are co-planar with the actual target area, the screen.)

I appreciate you clarifying, but no I am not conflating the corner pin region with the tracking region. Here is an example of a tracker region:

Tracking region

corner aspect05.png (526.37 KiB) Viewed 2166 times

And its corresponding corner pin region:

Corner pin region

corner aspect06.png (814.14 KiB) Viewed 2166 times

And yes, I do realize that I could theoretically just place the corners of the corner pin exactly where I want the edges of the composite to be, skipping the whole transform step altogether, but there are two reasons that this isn't any better of an option. First, I can't eyeball the points in such a way that I would actually get the aspect ratio for the image any more correct than how I'm already doing it. And second, I would have even less information to help me get the correct angle for the plane than I do already. In this example, I know the left vertical line is perpendicular to the line that the top line follows, and I know I can follow the ground with the bottom line, which means I really only have to eyeball one of the four points, which is good enough when using the grid. If I tried to put the corner pin exactly where the image is going, I'd be less accurate with the perspective angle, and I still wouldn't be any closer to a correct aspect ratio for the comp.

Hopefully that helps explain what I'm struggling with a little more.

Just follow Bryan's advice (and mine) about fspy and eyeballing and you should be on your way in no time.

Hmm, yeah, fSpy does exactly the math I'm talking about. Is it unrealistic to put in a feature request to have that type of functionality added to the planar tracker? I know that just because fSpy is open source doesn't mean it's as simple as copying and pasting a block of code, but it still feels like something you shouldn't have to use an entirely different piece of software for. Maybe I don't know enough about what I'm talking about. Thanks for all your help though.

It's absolutely a reasonable feature request! Although it's even more relevant to the 3D CameraTracker than PlanarTracker. Some tools to assist in estimating focal length and scene orientation would be very welcome. And lens distortion.