3D Camera Tracker

Hi Everyone,

This is my first post here and I'm pretty new to VFX and 3D Compositing. I;m working on a teaser for my feature film and I'm have a long track I'm trying to complete in the fusion tab of Resolve 15 and nothing I seem to to can get a good solve.. The shot is about 2000 frames shot at 60p for slow motion. The track itself is pretty straight forward and smooth tracking backwards from a person at the door of a house to revel the house in it's entirety. The shot was captured with a glidecam and other than a small amount of tilt part way through the shot I didn't think it would be that hard to track and solve.

When running the solve command it take quite a while and even when paring down the number of fully weighted track points to only those with good long tracks and setting the weight to 0 for those with a track error above 0.15 my resulting solve error is anywhere from 6 to 35. The path itself looks pretty good until about half way through then it jumps around very erratically. In the section of the track that is "sort of" working the camera position appears very jerky and not at all as smooth as the actual camera movement.

I'm sure there's something I'm doing wrong as the track points themselves appear huge in comparison to the camera track. Also the movement path seems shorter than the path that was actually traversed.

Any help I can get with the initial setup of the track (camera settings, lens settings, distortion correction, etc) and any suggestions to improve the track would be greatly appreciated. I know a fully featured tracker like synth eyes or the tracker in NukeX would be better suited but until we get the budget in place for the feature purchasing those is out of the question.

Camera: GH5
Lens: Rokinon Cine DS 24mm

Show the footage if possible. Symptoms indicate wrong fov which itself indicates not enough spread of tracking points for proper deduction of fov.

I'd forget NukeX tracker, it has no qualitative advantage over Fusion one, they are both meant for mostly simple shots that allow autotracking. Get Syntheyes or some other proper matchmoving software for more control and difficult shots. But also keep in mind matchmoving is an art, not just pushing buttons, so any soft is only as good as its operator

Thanks!

I'll post the footage here later today with some screenshots too. I'm looking for a place to start and some pointers to help pull a decent track. I'm taking the next 6-9 months to learn as much as I can and test a number of vfx scenarios in preparation for my feature next year.

Make sure you've set the Aperture to match the correct sensor size for your camera. The GH5 has a film back of 17.3 mm x 13 mm, according to http://vfxcamdb.com/panasonic-lumix-gh5/ (An indispensable site for tracking).

Make sure the tracker is picking points with sufficient parallax differences. Ideally, you want at least two points each in the foreground, midground, and background. Sometimes you'll spot an ideal feature that the tracker just won't pick up on. When that happens, you can create a synthetic tracking marker by point tracking the spot and adding a small, high contrast shape that the tracker is more likely to see. Hopefully one day we'll get proper supervised tracking, but until then this is a reasonable way to do a user track.

The freak-out in the middle probably happens when there is an insufficient number of overlapping trackers, or one of them drifts just a little too far off of its intended target. Adding a couple of additional synthetic markers across that spot can help. Depth of field can also be a problem because the tracker will tend to pick up on a false edge, and as something moves in and out of focus, it appears to change in size, which can fool the tracker. Again, a synthetic marker covering up the out of focus object and giving a high-contrast point in the center of the feature can solve your problem.

Rolling shutter probably isn't too much of a problem if you're shooting at 60p, but if there's very a fast turn or something you might be getting some shear. I seem to recall that there's a fuse out there somewhere that can perform rolling shutter compensation. I've never tried it with actual footage, so I can't say whether it works well or not.

You can save yourself some time by setting the seed frames yourself. If you look at the output in the console, you'll likely see that the solve takes one or two seconds at most, but the automatic selection of seed frames can take several minutes (I've seen it go as long as 45). The seed frames provide the initial solution. They should have as many trackers in common as possible but still have as much parallax difference as you can get within that constraint.

You may find that there are some tracks that look rock-solid, but they're actually tracking an intersection between a foreground and background detail. For instance, in this photo, the circled spot is quite likely to be picked up by the tracker, but it's a bad feature because as the camera moves, that intersection will slide up and down the table leg:

trackingproblem.jpeg (183.9 KiB) Viewed 11592 times

It takes scrubbing back and forth through your tracked footage several times to spot all of these things, and it's one reason why fewer really good tracks is better than lots of questionable ones—you don't want to spend your time trying to sort through 150 tracking points looking for intersections. Fusion's CameraTracker actually does quite a nice job of rejecting points like this automatically, but it will never find all of them, and sometimes it will get fooled into thinking the intersections are the good points and use them as justification to reject other trackers that actually are useful. When that happens, use a mask on the offending intersection to prevent the tracker from grabbing points there.

Thank you so much for the advice and suggestions!

I fought with the tracker for a while last night trying as best I could to eliminate problematic tracks and anything that seemed to be grabbing and edge with parallax between foreground and background elements. I've also manually deleted all the trackers that caught my actor moving with the camera at the beginning of the track so as not to confuse Fusion regarding the plane of motion. I also manually deleted all the points from the window reflections so as not to cause depth and motion confusion.

I'm not sure if I'm having the tracker grab too many points to track or possibly not paring down which tracks to use. What is a good number to solve from? When I track I get anywhere between 20,000 and 400,000 points. I've tried knocking that down to 5000, 2000, 600, 300, and as low as 150 but I never get a decent solve. I've also tried both deleting bad tracks and setting their solve weight to 0 and the "good" tracks to 1. I'm thinking about setting good tracks to 1 and exceptional tracks to maybe 2.

I tried manually selecting the tracks that looked good and solid then setting only their solve weight to 1 but that didn't work either. I seem to be consistently getting a solve error or 50-65 which is atrocious.

Also I have set the camera parameters for the GH5 sensor, even accounting for the 1.85:1 aspect ration when calculating the sensor height. I have set the lens parameter to 24mm matching the lens used. Do I need to apply a distortion corrector node before the track? I notice that the edge distortion is affecting the tracks around the outside edge.

I have also removed any post stabilization to be sure it's not affecting the track. IBS was enabled on the GH5 but since this was on a glidecam I don't think it should be affecting the track that much

Any advice regarding best practices would be awesome as I figure I'm missing something simple here.

I have shared out a copy of the rough grade I did before starting this whole 3d camera tracking adventure, in the fusion tab I'm working with the ungraded full 16:9 footage. Maybe there's something I'm not noticing that's through this whole thing out of whack?

https://drive.google.com/file/d/1gwuL4L ... sp=sharing

Greg H.

As an addendum to my last post..

This was shot using a metabones speedbooster ultra 0.71. Should I modify the sensor size to account for the focal reducer in the camera info tab prior to running the solve?

Thanks!

Yes to distortion correction--that will certainly affect your results.

The speedbooster... Probably, yes. If it modifies the field of view, that would definitely affect your results. I have never dealt with such a device, so I am unsure exactly how it interacts with the sensor size and focal length. In reality, neither of those numbers are what is important. It's the field of view that you actually need.

It's also worth mentioning that Fusion's Camera Tracker can give wildly different accuracy results after only minor changes to its parameters.

As to how many trackers are enough, in a perfect scenario you can get a solve with as few as eight if they're properly distributed. Adding more reduces random errors since they'll average out to a result that is more likely to be accurate.

You may also find that there are portions of your shot that solve well, but there's a portion in the middle where the tracker has trouble. If that's the case, you can solve each side separately, manually animate over the troublesome part, and then line the three resulting cameras up using Transform3D nodes. You'd then have to cross dissolve between the renders for the three cameras. That's obviously an effort-intensive route to take, but sometimes it's the only way to get an acceptable result.

From the looks of this shot I'd say to get a nice continuous solve you would need survey data and supervised solving. You probably don't have any survey data?

With no additional data I'd solve it in parts, and probably use mostly manual tracks, well selected, well tracked and well distributed spatially.

For survey data, easiest way is to shoot a good selection of photos with a fixed wide-angle lens, also shoot a distortion grid for this lens, undistort the photos and manually solve for all camera positions. With solve you get a set of 3D points what you can in turn use for constraining your movie shot solve. But to do all this you need proper matchmoving software.

I'm no expert at this but I have found that the biggest keys to success/failure with camera tracking is getting the camera parameters *dead* right. Even the smallest change in focal length, apperture, etc can make enormous differences to the quality of the solve. I recall that in some bits of footage, I've gone from a solve error of 30+ down to less than 1 with a single (minor) change to these parameters.

Either I overlooked or nobody asked about it, but: did you mask out all the people before starting the track?
And in addition to this: why are you tracking the whole shot? What is it that you're trying to add in 3D that you need the camera track for? (I'm not saying you shouldn't track the whole shot, I'm just curious)

Thanks again for the suggestions. As far as survey data I do have a number of pictures of the location from a few different angles.

I'm not looking to track the entire shot, only from 1:25 onward. The idea is to add in fire through the trees and to the side of the house. The elements themselves will be mostly 2d however their size and position would need to change as the camera moves away and the parallax would need to be consistent with the position of the actual element they are attached to.

I have manually deleted all the tracks that attach to the actress and the windows since there are reflections that will throw off the solve.

I've now corrected the lens distortion and set the sensor size correctly for a GH5 with the speedbooster 0.64x which I confirmed was used when we shot.

I'm hoping these changes lead to a more useable track.

Finally got the chance to look at your footage. You'll want to throw away any trackers on the trees, too. There's just a little bit of a breeze, so any movement there will give you trouble. It's also all but impossible to separate foreground and background motion from the overlapping details. I mean, if you hate yourself you could curate those trackers, but I wouldn't bother with them.

I can see the lens distortion pretty clearly, so fixing that will certainly help. I don't know that it's necessary to shoot grids. You've got some nice vertical lines near the edge of screen at the head of the shot, and the focal length is constant, so you should be able to undistort without too much trouble.

Hopefully the film back correction will be the last piece of the puzzle you need to get it nailed down.

Constraining both filmback size and focal length is a bad idea unless you know exactly what the values are. And no, exact values are not the ones printed on lens or in spec sheets, but the ones you get from solving a very easily solvable scene with very good spatial layout of points and preferably close to 90 degree angle between at least two camera positions. Without exact values it is usually way better to let solver calculate focal length based on set filmback size.

I just wanted to provide an update for everyone who's been so helpful thus far.

This evening I've been solving and resolving the track killing off obviously erroneous points from the point cloud each time. It seems like many of the tracks were really really bad which was completely throwing off the scene size and camera travel distance. By iteratively eliminating points that fall much too far away from the camera I have been able to bring the solve error from 105 to 35 and now down to 9.75.

I will continue to pare down the points until I can get a good camera travel distance and a solve below 0.5 and see where that gets me for camera tracking.

Thank you again for all the advice, this has been quite the learning experience.

It's nice to see you're making progress, but you're better off masking out the obviously moving parts of your image before tracking, because jus throwing away tracks doesn't always result in a better track.
If you mask out in advance Fusion will try to add more trackers in places where you want them.
It's definitely worth giving it a try.

Hi again, 20+ hours in and I'm still fighting with the tracker. I've masked off all the distant/moving trees, the actress while she's moving in the frame (removed once she's still) and lastly all the reflective surfaces.

I'm not really finding that this helps the solver. For some reason the camera basely moves on it's path then jumps around erratically with no rhyme or reason. The perspective tens to change wildly too which is very confusing..

I'm wondering if there just aren't enough consistent tracking points for the tracker to solve. I know for sure that there are a number that stay ion place for a good 150 frames. The total track at this point is about 1500 frames. Unfortunately there are no tracks that last the entire duration of the track but my understanding is that this should cause the solve to fail as long as there a few good long tracks at different distances. I have calculated the exact sensor size and corrected for lens distortion before tracking.

It's almost as if it's adding points in the same screen location but wildly different z depths. This causes the camera to lock it's position since it cannot calculate the appropriate z depth to figure out where to move. I wish there was a way to align groups of points that occupy the same z depth but that would require supervised tracking which is beyond the scope of the fusion tracker. I'll post a few screenshots of my settings and my resulting camera paths, point could and 2d point mapping. I still feel like I'm missing something fundamental about setting the track up.. That or this sequence is just beyond the scope of fusions tracker.

Here are a few screenshots showing some of my settings and resulting tracking. Any advise is appreciated. More screens to follow.

Far view of camera path

far.jpg (316.65 KiB) Viewed 11329 times

Close view of camera path

close.jpg (370.6 KiB) Viewed 11329 times

[attachment=0]point cloud far.jpg[/attachment

More screens.

Close view of point cloud

point cloud close.jpg (132 KiB) Viewed 11327 times

Camera tab in tracker node

tracker(camera).jpg (62.63 KiB) Viewed 11327 times

Solve tab in tracker node

tracker(solve).jpg (67.22 KiB) Viewed 11327 times

I ran your shot through Syntheyes without any masking etc and it couldn't come up with a solution.
I didn't try it with masking, because...well, that's because it's your shot
but it is definitely a shot that would benefit from (if not require) supervised tracking. Also: like was mentioned in previous posts, if you're not entirely sure about stuff like sensor size you're usually better off leaving that to the solver.

Also the size of your solve isn't as important as the angle of your solve. In that sense 3D comping is similar to old fashion trick photograpy: as long as the angles line up, the images will as well. Like it does with foreground miniatures, glass painted matte paintings etc.

I pulled it down to try it, too, but the version we see here has too many temporal compression artifacts. Some details hold still for several frames while the rest of the image slides around them, then they jump forward on the next i frame. You did say that this was the post-grade footage, and you're working on (hopefully) the first-generation. If there were similar compression artifacts in the original footage, though, then there's little chance any automatic tracking is going to work at all. That is yet another example of why video formats are not used in VFX.

Yep, my trackers looked like a bunch of ants crawling all over the place, that footage is a bit nasty.

To be honest I think you may get away with using 2D/Planar trackers for each fire element you are adding. The great thing about fire is that it randomly changes so perspective shifts, small drifts in accuracy may not be a problem. The tracker will take care off scale, translation and rotation, just as long as you can approximate the tracker to an item of the same depth and position as the fire element.

I have had to do a few shots with 2 1/2 D tracking and the reality is they looked fine and I got them done in a lot less time that I wasted 3D tracking to get an unusable track.

This shot is also a good example of where supervised tracking would be beneficial.

Thanks again everyone. I was afraid this was a codec/footage issue.

The only 60p camera I had access to was a gh5. I guess I should have used my blackmagic camera and just shot Raw 24p. We shot 1080 200mbit but I guess the compression was just too much for the features as they drift away from the camera. I'll give the 2d planar tracker a try and use some stock fire footage with transparency. I'll try tracking some of my raw footage from other projects to see if the tracker can at least give me something closer.

Good news is I've got lots of raw footage I can test with. Also the feature I'm hoping to make will be cinema DNG or blackmagic raw.

Ok, managed to get a Solve Error at 0.67 after 2 passes. I only used 1300-2200 frame range as I couldn't be bothered tracking the whole thing.

Estimated the focal length by eye to start at 24mm and set refine focal length and lens parameters.

Didn't bother masking just roughed out most bad trackers by eye, still a lot of bad tracks there though (Note: Any tracker on those black rails will cause big problems so remove). I think the trees are ok as they are not moving too much and otherwise most information is sitting on a single plane.

Also don't forget the importance of the seed frames, they will play a big part in getting a good solve (I used to auto which worked this time, often I will set manually).

Link to comp:

https://www.dropbox.com/s/rtwgkkdhucddo ... .comp?dl=0

I notice you've got windows in that scene... I've always found it far more difficult to get a good track when the scene has windows with reflections. Lots of masking required because the reflections *really* confuse the tracker.

Thanks Kel!


I was working with the footage range 2527-3965. I have definitely masked out the windows and isolated the house from the moving trees. I'm running another solve with a higher track count paring them down to about 1000 points. I'm hoping with a couple clean up passes I'll be able to get something useable.

To answer some other questions I am using the 1:85-1 original v-log footage shot at h264 all-I 200mbit. I'm hoping this all comes down to grabbing enough points then paring them down further and further.

I also tried the planar tracker an might go that route in the end.

Thanks again!

Hello again. I have some great news!

I think I've finally got the solve working and I know what my original issues was and I feel extraordinarily dumb.

When adding trackers to the solve I thought picking as few "1" weighted trackers would yield a more accurate result but I could not have been more wrong... I fed the solver 3000 points from which to solve the path and got a solve error of 4.26. We're not there yet but with some refinements and possibly more data points I'm pretty sure I'll get a good enough result.

Thanks again everyone for all your help and the extra effort and explanation provided. If anyone has some suggestions about training for Fusion I would love to continue education myself while I work through a few more test shots.

Greg H.

I just started writing a Tool script that will allow users to copy over trackers from a 2D tracker into the camera tracker to allow user tracking (or if this is possible someone please let me know ASAP). Will post on WSL when I'm done.

Kel, that kind of functionality would be fantastic and would help bring the fusion trackers functionality closer to what's available in Nuke X. If there was a way to map parts of the video image to the individual tracking points or a way to create a polygon mesh to more easily integrate 3d elements into the scene that would be awesome too.

From everything I see the back end ability to add many of the features that set dedicated trackers like syntheyes or pftrack apart is there in Fusion. If anyone from blackmagic is listening more manual tracking options for the camera tracker would be amazing!

From everything I see the back end ability to add many of the features that set dedicated trackers like syntheyes or pftrack apart is there in Fusion. If anyone from blackmagic is listening more manual tracking options for the camera tracker would be amazing!

There is a lot more to dedicated matchmoving soft than adding manual feature tracks and a few features here and there. Syntheyes for example (don't know about pftrack, haven't used that) has a very robust and fast solver that is a great deal of work in itself. The whole constraints system which is directly related to solver is huge undertaking and essential to any decent matchmoving soft.

From watching a number of Syntheyes tutorials I'll definitely be investing in a copy before shooting my feature. I can already think of a number of instances where it would be far superior to the Fusion tracker.

Thanks for the info again, I'm trying to learn as much as I can about not only the practical application of the technology but the underlying math/physics calculations that make it all possible.

There is a lot more to dedicated matchmoving soft than adding manual feature tracks and a few features here and there. Syntheyes for example (don't know about pftrack, haven't used that) has a very robust and fast solver that is a great deal of work in itself. The whole constraints system which is directly related to solver is huge undertaking and essential to any decent matchmoving soft.

PFTrack is my weapon of choice, and it's much the same. Although it does a reasonable job of getting a solve without geometric constraints on many shots.

Fusion's tracker is nice for reasonably simple tracks where absolute precision can be sacrificed. There is a certain advantage to having the track right there in the software where you're going to use it because you have more tools available to save a marginal track. If I can get a track to within a certain degree of usefulness, I'll make up the difference with a Transform3D and maybe a couple of grid warps. A "cheat" that gets you there in two hours is far superior to a technically perfect solve that takes two days. As long as someone else in the pipeline isn't using the track for another purpose, at any rate.

Out of the two hundred or so tracks I've done in production since Fusion 9 was released, though, I've only used CameraTracker thrice. Once as a quick bridge between two other cameras where I just needed a quick-and-dirty solution really fast. Once in a shot where I was just adding rain and could get away with something that drifted a little. Although in that latter case I actually got a rock-solid result that was just as good as any result I might have gotten from PFTrack. The third attempt was one that I probably should never have tried in Fusion, and it failed.

It's a nice start, but it definitely has a way to go before I'd consider it a useful enough tool to be worth the admission price. I certainly wouldn't buy Fusion Studio on the basis of CameraTracker alone.

Anyone have any experience with using Blender as a camera tracker?

Well despite it being

There is a lot more to dedicated matchmoving soft than adding manual feature tracks and a few features here and there. Syntheyes for example (don't know about pftrack, haven't used that) has a very robust and fast solver that is a great deal of work in itself. The whole constraints system which is directly related to solver is huge undertaking and essential to any decent matchmoving soft.

You are right Hendrik, but user tracks will help my workflow and they are something you can do after the event when constraint information, survey data is not available. I have had shots where I got a track by using only User tracks as the auto tracking just gave me garbage on every tracker I used.

Anyway for what its worth here is a Beta of the script that will add info from a Tracker into a Camera Tracker.

https://www.steakunderwater.com/wesuckl ... f=6&t=2602

User tracks are great, what I'm saying is that user tracks is not what separates Fusions or NukeX camera tracker from dedicated matchmoving soft. User tracks are the basic bread and butter of matchmoving. Stuff starts to get nasty when even very carefully chosen feature tracks, right fov etc still give you no solve or bad solve. Then what? Then you must start forcing some additional logic onto solver to massage the solution into more reasonable form. In SE for example you can feed manually animated curves into any parameter (camera position, fov, rotation on Z axis etc) to act as soft or hard constraint.

Really? That's a nice feature; I wish PFTrack would do it. At best you can give it a manually animated camera as a hint, but it's an all-or-nothing thing, and there's no way to tell it how confident you are in the estimate. I had one just today where I wished I could lock x and z rotation and smooth only on z translation.

Really? That's a nice feature; I wish PFTrack would do it. At best you can give it a manually animated camera as a hint, but it's an all-or-nothing thing, and there's no way to tell it how confident you are in the estimate. I had one just today where I wished I could lock x and z rotation and smooth only on z translation.

you can use syntheyes soft or hard constraint to do it via this dialog box