Planar Tracker Fine Tuning?

Hi Gang!

So I am tracking a ridiculously simple portion of a building as I am tilting down the camera. From somewhere in the middle the tracking region (a rectangle in my case) starts to shift ever so slightly so by the time I reach the end of the clip the tracker shifted quite a bit.
My question is - how do I fine tune the keyframes on the path of the tracker to adjust the region properly?

Thanks!
G

Hi All -

I thought of explaining this a bit better with images. My planar tracker starts out nicely until about the middle of the clip when the tracking goes off-track (that was a pun), and by the end shifts. I think as one who comes from an After Effect background, I am looking for a way to manually adjust the keyframes of the tracker path. I saw that is possible with the regular tracker but not sure how to do that with the planar tracker.
Or am I using the wrong tracker for this? It's a simple tilt camera movement, nothing intricate.

1. Start of clip:


2. End of clip:


My nodes:


Thanks a lot!