Solving Facial Performance Capture using Fusion 9

Good evening. This is my first post and after someone told me about Fusion and how I could use it for my projects, I decided to look into it. My name is Kashif and I am an indie 3D artist located in Boston and I have been working on a project that involves the use of motion capture, particularly using match moving software to solve facial animation data. I have looked at other programs like Autodesk Matchmover (which is great, but it doesn't work with the Modo workflow), Nuke (too expensive and not feasible for an indie), PFTrack (which is another great program. Was able to VERY quickly do a 3D solve of my facial capture, but it is very costly for an indie), and I even looked at Syntheyes (great program, but the UI is hideous and it requires too much manual setup to do my facial capture solves. So now I am looking at Fusion 9 (I do understand that I will need Fusion Studio in order to do tracking)

Such as, retargeting tracking data to these points.:

Facial Capture Layout

Screenshot 2018-11-04 22.25.34.png (562.54 KiB) Viewed 3273 times

Screenshot 2018-11-04 22.26.02.png (558.47 KiB) Viewed 3273 times

Screenshot 2018-11-04 22.39.19.png (432.99 KiB) Viewed 3273 times

What I would like to know is if fusion would be able to take two pieces of footage that I record from my custom head rig (like they use in Film and Games) and track the data to be exported to a 3D package like Modo...? Maybe with a motion capture node or an equivalent that allows me to track the dots from stationary cameras to build a 3D solve.

Thank you very much.

Fusion has a camera tracker, but it is not as sophisticated as a real matchmove software.
For example right out of the box it doesn't even allow you to use user tracks, only auto travjing

Perhaps that can be a feature request

You mentioned that nuke would be too expensive for you. Have you looked at athera.com? A nuke in the cloud costs a couple of hundred dollars for a 30 day period, plus a couple of dollars an hour for actual usage. Is that more accessible?
Dc.

Nuke is useless for what he wants to do. Its camera tracker is similar to Fusions and can't be used for multi-camera facial solves.

Nuke is useless for what he wants to do. Its camera tracker is similar to Fusions and can't be used for multi-camera facial solves.

This course details a nukex to Maya workflow. The OP wants to go to modo, which is also a foundry product so you would think there is a route there somewhere.

https://www.pluralsight.com/courses/nuk ... ction-2545

Or perhaps with the face-tracker plug in from KeenTools, if it allows its mesh data to be exported.
https://www.keentools.io/

Perhaps that can be a feature request

In the meantime I created a script that allows you to copy 2D trackers over to the Camera Tracker here:

https://www.steakunderwater.com/wesuckl ... f=6&t=2602

I'll do some looking into that. I'm using three cameras to solve my performance. Perfect example is the process used in games like Infamous Second Son. I was also trying syntheyes, but it took too much manual setup

Nuke is useless for what he wants to do. Its camera tracker is similar to Fusions and can't be used for multi-camera facial solves.

This course details a nukex to Maya workflow. The OP wants to go to modo, which is also a foundry product so you would think there is a route there somewhere.

https://www.pluralsight.com/courses/nuk ... ction-2545

Going to Modo is not the problem, problem is the existence of data to send in the firstplace. This course is about tracking head motion, not solving facial deformations from stereo footage that OP wants to do. It is a big leap from one to another and NukeX cam tracker does not allow multi-camera input for solving each feature individually, which is the basis for facial track. It might not seem like a big difference but it is. Object track is basically the same thing as camera track because relative motion is the same. In one case camera is moving and object is still, in another, object is moving and camera is still. It is easy to go from one to another. But facial rig needs all markers to be solved separately in 3D space, which can be done by either using more than one view or heavily constrained matchmove rig and solve.

And I made absolutely sure that all of the facial markers can be seen in both cameras... I'm planning on adding a third camera to serve as a potential witness and it will look at me dead center

Here is the inspiration for why I'm doing this

Unlike a 3D solve though I think this wouldn't be that difficult, as each points movement is totally independant of the others. If identical cameras are placed at the correct angles and trackers are not obscured you should be able to match one cameras undistorted x,y tracking coordinates with the 2nd cameras tracking y,z undistorted coordinates for each point. We used to use a similar technique to track the shooting camera if we were concerned about how much tracking info was in the shot. I am probably over simplifying this though.

I was looking for ways to do this. Both of my cameras can see the middle of the face... sadly I'm using fours cameras so EVERYTHING can be seen on the face. And the intended result is a 3d solve. Perfect example is Vicon Cara...

Using a 2d solve, though it works, results in parts of the face feeling flat. Currently, I have two action cameras and I can't track my face completely (I have markers on the side of my face that are obscured.)

In PFTrack as long as two cameras can see the markers, they will 3d solve... so I have two on the left, two on the right, and they all see the middle of the face. And with a user track and a mocap node, I can do all of this.

So I think a user track and a mocap node should be a feature. The price for Fusion 9 studio is great for indies.

Unlike a 3D solve though I think this wouldn't be that difficult, as each points movement is totally independant of the others. If identical cameras are placed at the correct angles and trackers are not obscured you should be able to match one cameras undistorted x,y tracking coordinates with the 2nd cameras tracking y,z undistorted coordinates for each point. We used to use a similar technique to track the shooting camera if we were concerned about how much tracking info was in the shot. I am probably over simplifying this though.

Basically yes, if you know the relative position of cameras, their view vectors and fov, then from the 2d location of feature in screen coordinates you can calculate its 3D position. But unless one is willing to do the math by scripting, it doesn't really help

Heh. Yeah, it's one of those problems that's simple, but not easy.

3DEqualizer has reasonably affordable weekly and monthly rental options, and a personal learning edition. You could learn the technique in the learning edition, then subscribe for a short time to get your actual production solves done and exported, then cancel. If this is something you think you'll only need to do a couple of times over a short period, that could be an option--you'd get the benefit of the powerful pro solution at an indie price, at the cost of not having a permanent license.

While the underlying tools are there in Fusion to allow a performance capture system to be built on it, the development cost would be prohibitive. You'd wind up either having to do it for the love of the project or charge $1500+ for a plug-in for $300 software, and I just don't see anybody willing to take that kind of a risk at the moment. BMD themselves have their hands full trying to wedge Fusion into Resolve, a project which is currently delaying the development of new features for Fusion. And honestly, the current CameraTracker is kind of half-baked as it is.

I can't imagine that... and I was reading a tutorial on mocap with Syntheyes and the setup is rather extreme where I have to calibrate the camera using some porcupine setup, solve it, then replace the footage with my own... I just don't see it happening with four cameras.

What this calibration gives is correct fov, nothing more. If you know it for your cameras (for any next shot), just use the values and set fov to fixed. In Syntheyes I'd prepare a template project with four cameras and then for any new shot just replace the footage. It isn't any harder than doing it somewhere else.

Also Syntheyes - and the guy creating it - are very good at what they do, so if their workflow consists of stripping naked and doing a dance in the full moon while calibrating your sensor and singing Y.M.C.A, I'd suggest following those steps before trying anything else.

What this calibration gives is correct fov, nothing more. If you know it for your cameras (for any next shot), just use the values and set fov to fixed. In Syntheyes I'd prepare a template project with four cameras and then for any new shot just replace the footage. It isn't any harder than doing it somewhere else.

True. And I guess one advantage is I can just set it up once...

Also Syntheyes - and the guy creating it - are very good at what they do, so if their workflow consists of stripping naked and doing a dance in the full moon while calibrating your sensor and singing Y.M.C.A, I'd suggest following those steps before trying anything else.