Feature request: Neural engine camera tracker

There's been a surge of neural engine/neural processing AI-enhanced tools that make life a lot easier. Things like keying without a green screen, EBsynth, Ember Gen etc. all that use some kind of neural processing to both speeds up the process and make it essentially one-click to result with just minor adjustments.

But there's still a big area that never gets attention and that is basically the most important part of VFX work and compositing: camera tracking.

I'm still waiting for SteadXP to mature enough to make a version of their box that works with high-end cinema for stabilization, but I'd also want them to release something that basically tracks the camera and gives you an FBX camera straight out of a shoot day. But since we don't have that, why isn't anyone working on a neural engine camera tracker? Something that basically uses far more advanced methods to calculate not only camera movements, but that can detect and remove the movement of actors and other stuff in a scene that doesn't relate to the camera move. Like, an inverted neural engine object tracker, instead, tracking everything else. As well as automatically figure out perspective, ground plane etc.

I'm checking research papers on a lot of interesting stuff that's coming, but none of them mentions a good camera tracker.

I would bet, that if Blackmagic released Fusion with a neural engine camera tracker that almost works as a one-click, it would be the go-to tool for the entire industry. It's such an untapped potential and would save the biggest time-consumer that we have at the moment.

I'm doing a lot of work in Blender, and that camera tracker is good but very bad at working without a lot of input. So I usually track cameras in Fusion and export the camera to Blender. But it's still so time-consuming and the result is almost always extremely off the mark.

So, if a neural engine camera tracker was the ONLY update coming in the next version of Fusion, that would be enough to justify a whole version beat. Basically, if Fusion had something like EBsynth, Runway, and a Camera tracker, it's game over for other compositing software. Doing a camera track, keying someone without a green screen, and adding features to that person in a matter of minutes rather than hours would revolutionize compositing workflows.

Here are the other tools as references.
https://runwayml.com/
https://ebsynth.com/

Flame got ML aided camera tracker:

I wouldn't get too excited about one-click solutions, they always only go that far. No-one will dump their software for something else for just one feature that might sometimes work well. Industry doesn't work like that, and 3DEqualizer specialists will not be suddenly out of jobs now that Flame as put it out. You can basically do everything that Runway and EBSynth do in comp today, and more, because you actually have control over it. Is it game over for Fusion now?

Flame got ML aided camera tracker:

I wouldn't get too excited about one-click solutions, they always only go that far. No-one will dump their software for something else for just one feature that might sometimes work well. Industry doesn't work like that, and 3DEqualizer specialists will not be suddenly out of jobs now that Flame as put it out. You can basically do everything that Runway and EBSynth do in comp today, and more, because you actually have control over it. Is it game over for Fusion now?

Have you tried Runway and EBSynth? What I mean with "one-click" is that they are essentially one-click, but you still have to comp and work with it afterward just like any other comp tool. What I mean by "one-click" is that there's no tracker management, manual perspective solving, and so on. There are thousands of things that go wrong every time you try to do a camera track and most of the time you need perfect material with good contrasts to be able to.

Can you do what Runway and EBSynth do? I'm not saying they are replacing Fusion, Flame or anything, but their kind process inside of something like Fusion would be extremely powerful. Adobe is pretty close with their new version of Rotobrush, but the point is that these tools in sync with regular comp work remove an extreme amount of time wasted, especially when the material isn't optimal.

When you say "they always only go that far", that's a statement of what has been, not what is around the corner. The surge in machine learning tools has just begun to pop up, we haven't really had them in the way they exist now and they are getting better in each iteration. It's not about people being out of a job, they will learn new workflows based on technological improvements, but they won't be hired to do camera tracking for days and days if we have tools that do a first track in "one-click" and then there are just minor adjustments for getting a perfect camera track.

The feature request is to improve the camera tracker into a neural engine, machine learning camera tracker that can do a complete track and solve with machine learning instead of the tedious ways we're doing it now.

Like the fact that we can isolate people in Resolve Color, but not able to use that roto in Fusion is... just plain dumb.

Flame is doing it right, you don't need to hire more people to do camera tracking for comp if you have those tools like that. However, from what I understand, Autodesk uses server ML? And all of that costs a lot of money. But there are ways to utilize the GPU architecture of new GPUs like Nvidia's RTX series to handle the machine learning aspects. That is what can make something cost-effective but still use ML techniques.

This is what I want Fusion to be. Being a swiss-knife company I cannot justify the cost of Flame so I'm stuck with Fusion and I hope Blackmagic takes notice and improve Fusion into this new age of ML technologies, otherwise it's gonna die.

Like... check out the best value price *facepalm*

For me, it's like, wake me up when there's a reasonably priced tool for small businesses like my own. The tools I refer to are what's coming and getting that kind of power in Fusion and comparing it to the price of Flame becomes ridiculous. Already using Blender for a lot of effects and it's free. There's no business to be made if I put down money on Flame, because the money gets put into good Red cameras, lighting equipment, sound, editing, grading etc. Flame is for enterprises only. Fusion is basically free, or not even comparable in price to Flame. But it still has the potential of being a competitor. Just like Blender is getting used more and more in serious VFX companies.

Autodesk is king because they have the funding to make those tools, but at the same time, they're gonna be killed off just like the extreme cost of Resolve got cut down to basically free the further we get advanced tools democratized.

I'd just wish Fusion was the frontrunner of that democratization, just like Resolve is pretty much destroying the competition due to all the new generations of colorists jumping onto Resolve because it's basically the most logical option to use. Blackmagic is playing the long game with Resolve, when older colorists retire and new blood comes in, they're gonna use Resolve Color, that's where things are heading.

So, Fusion should do so as well, get some machine learning tools in there and let's go.

Time will tell which software, approach or "philosophy" is around after a while. Only Fusion product manager knows what is up with Fu development. Wake me up when BMD finds out who Fu product manager actually is. I'd love to see Fu kick competition in the nuts because it re-animates the field, but for it to happen, there must be a lot happening with Fu first.

Runways roto is comparable quality wise to magic mask, rotobot and other segmentation models (mask rCNN). And you can train an even more detailed specific roto model in Nuke using copycat. There was a funny demo feeding rotobot masks to copycat to get an even better roto out of footage with no manual work whatsoever.

Their other ML tools use off the shelf models, from the looks of it I'd say face blends are StyleGan2 based, object blends are also some GAN models.

EBSynth is motion vectors + keyframes, essentially what smartvector toolset in Nuke has been doing for a few years now.

When you say "they always only go that far", that's a statement of what has been, not what is around the corner.

The law of rendering does not agree. Basically it states that expectations always run ahead of possibilities.

The law of rendering does not agree. Basically it states that expectations always run ahead of possibilities.

Yes, but "one-click" solutions does not also mean you push a button and all the creative work gets done as well. "One-click" solutions can be a one-click of one area. Flame has a one-click to track and solve, the rest is creative work. This is the "one-click" I mean and the possibilities are already there, just depends on how it's being implemented.

At the moment, both Fusion and Fairlight feel like wastelands. It's like there's nothing there or happening to it. Fairlight is nowhere near what you can do in something like Adobe Audition. The sound effect search is broken and really bad.

Most attention goes into edit and color. But imagine if the same effort went into Fusion and Fairlight. Autodesk is already losing a generation of 3D artists to Blender, as well as Unreal engine. Just as new generations ditch Baselight and Nukoda in favor of Resolve.

Who wants to be the frontrunner of cheaper software for compositing? After Effects still doesn't have any node-based workflows so they're never going to be able to really take over before they do something drastic like that. Everyone who's serious about compositing will eventually go into node-based workflows, so either Blender gets their compositing into a very serious shape, or Fusion does it. That's where things are heading. If Blackmagic doesn't take this seriously, then Fusion will be dead weight in their DaVinci package.

As someone who does a lot of camera matchmove (PFTrack), I agree that it's an area where machine learning could improve things. But I don't know that I'd want a one-click process. I want ML-powered object, perspective, and lens recognition, still guided by user input. It is so frustrating to have an automated tool give you 95% but then be completely useless because it's a black box that can't be tweaked (looking at you PlanarTracker!)

Fusion already has a very powerful flowgraph that can handle arbitrary datatypes. If CameraTracker took advantage of that, breaking out the tracker and giving us multiple solvers—lens modeling, camera solver, perspective alignment, image-based modeling—then some ML tools could be inserted where they're needed to assist the artist. And they could be developed piecemeal instead of needing to get everything done all at once.

User controllable flexible toolsets is where it should head, I agree. Easier said than done though, it is one thing to give users a pretrained model and big button to push, it is quite a different play to introduce actual configurability.

This article shares some interesting insights into area from competitors perspective. Afaik copycat is currently the only artist friendly (doesn’t need antics with pytorch/tensorcore) solution for training a generic image to image transfer model with high enough precision for actual comp usage (32bit float data): https://www.fxguide.com/fxfeatured/copy ... g-in-nuke/

As someone who does a lot of camera matchmove (PFTrack), I agree that it's an area where machine learning could improve things. But I don't know that I'd want a one-click process. I want ML-powered object, perspective, and lens recognition, still guided by user input. It is so frustrating to have an automated tool give you 95% but then be completely useless because it's a black box that can't be tweaked (looking at you PlanarTracker!)

Fusion already has a very powerful flowgraph that can handle arbitrary datatypes. If CameraTracker took advantage of that, breaking out the tracker and giving us multiple solvers—lens modeling, camera solver, perspective alignment, image-based modeling—then some ML tools could be inserted where they're needed to assist the artist. And they could be developed piecemeal instead of needing to get everything done all at once.

I'm all for being able to tweak things, it's the one thing I hate about the warp stabilizer in AE, it's damn good, but when you get errors or need to tweak something you're pretty much screwed.

A one-click version should track and solve everything as a first step. If 90% of the camera tracking situations are about getting tracking points nailed to the features of the footage, get the lens and perspective data, floor and the camera perfectly moving throughout the shot so you can just start comping in objects in the scene, then that should be the main part. Then if you need tweaking you should be able to.

But I think most are afraid of "one-click" because the general experience is that most such solutions are useless and a marketing gimmick. But the ones surfacing right now aren't, they're truly one-click with some very minor adjustments. It's like when window tracking for color grading became really good, it radically improved the speed at which you could do secondaries fast. The Flame one is a good indicator of how fast it becomes.

Combine that with someone making a hardware box for cameras to feature timecode-synced camera tracking in real-time with the footage and you pretty much get days of "solving" stuff in a couple of minutes and can start working on the actual creative stuff.

The amount of times when I just need a camera tracker solved and ready is far more often than I need to have some edge case control over something specific.

Copycat looks like a killer feature, but that is why Nuke is the go to for VFX.

Agree on planar tracker , the latest upgrade for fusion was to change computing from CPU to GPU.

Resolve has a cloud tracker in color but its data also cant be adjusted or used in fusion

Overall fusion didn't get any new tool. To me point and planar trackers look old ( but it works ) and paint looks ancient ( and it works ) needs more features, lets build more features in there plz ?

Features that its getting seem obvious and should have been added as point release not the yearly release , feels like fusion team is small where the edit and color teams at BMD are huge.

Black Magic seems to be focusing on editing in resolve ( as we have seen in the latest presentation for R.17, which majority for devoted to cut page ) fusion got some love here and there but no new tools ( you can hear audio in fusion now ).

Obviously you guys spoke about the price.
Well its free but I would love to see more time/money spend on building this software.

A one-click version should track and solve everything as a first step. If 90% of the camera tracking situations are about getting tracking points nailed to the features of the footage, get the lens and perspective data, floor and the camera perfectly moving throughout the shot so you can just start comping in objects in the scene, then that should be the main part. Then if you need tweaking you should be able to.

If it was separated into logical steps with manual override possibility it would be great. Given that the generic steps of matchmove are what you outlined, I'd first concentrate on the more time consuming parts. 2d feature tracks and elimination of outliers by discarding moving objects, reflections and other crap should be the main thing, because this is the foundation of everything else. Scene alignment and scaling based on object detection, what is ground and how big something should be (roughly) would be handy for quick work. And then doing a control step of whether the solve actually makes sense in context of scene, some kind of logic test. Not sure stuff like lens distortion estimation benefits much from ML tools, but maybe it does.

Not sure stuff like lens distortion estimation benefits much from ML tools, but maybe it does.

I'm not really "up" on ML techniques, but from what I understand of it, a neural network should be able to learn how to recognize both stretching on a moving shot and curved lines that should be straight, then select an undistortion profile to fix those things. The area where I currently have the most trouble is varying distortion due to lens breathing on the anamorphics that are so popular right now. If I get a rack focus, I usually have to outsource it to a 3DE artist because I just don't have the tools to handle it. A net that could manage that task alone would be worth a lot to me.

I don't know how to use Syntheyes well enough, but I think the most recent version has some sort of machine learning built in, and it also comes with Synthia that allows for a limited form of 'human like' interaction and it also comes with a bit of a modular approach similar to what everyone here has been talking about. I think you can create some sort of node like construction where every 'node' (for lack of a better word) takes care of a different step so it can be tailored to a specific problem.
This is all very complicated for me, since I'm no match mover and no programmer, but it has a special skill set for challenging shots.

@bryan: Syntheyes 1903 - anamorphic lens support including rack focus

I need to give Syntheyes another look, for sure. The last time I was in a position to make recommendations about tracking software, it didn't yet have geometry tracking, so it wasn't useful to us. That's no longer the case, but we're still locked in to PFTrack at work.

Ideally I'd like to be using 3DEqualizer, but the price tag on that is big yikes!

ML features in SynthEyes are currently for feature detection it seems, haven't tried newer versions, I must check if my license allows any of them (I always forget to update SE). Synthia is pretty old already but not sure how useful it is, interesting feature anyway. As I understand it, it is kind of a soft grammar analyzer that tries to convert somewhat random human sentences to SE commands. Nodes are a separate thing in SE, it is called Phase system and somewhat similar to PFTrack graph (afaik, haven't used PFTrack and don't use Phases either).

Phases, that's the thing I was referring to. Thanks, Hendrik!

As far as I know there is no syntheses forum so I'll ask here. Did anybody use the ML features in Syntheses and can tell how useful they are? I tried using them but seems like they don't run on my RTX 3070. I assume because Syntheses doesn't use the newest tensor flow library.
But I didn't want to write Russ about it yet because I have no idea if I would actually use it (and have no time right now for bug reports that don't actually affect me)

If I remember correctly either from the release notes or the manual, by default the ML is CPU only because of its download size. If you want to use NVIDIA specific code you have to install something seperately. I can't remember exactly, but it's somewhere on the site.

Yeah I did that. But I don't want to reinstall Syntheyes to get the original, overwritten CPU ML files back. I have to tackle that issue eventually, but now is not the time I just have fear of missing out regarding the ML tools in there, but looking at other ML software, I would not want to run this on a cpu.