Something like copycat Ai of Nuke?

Recently, I saw many interesting thing about nuke on copycat node function, it’s very interesting use of artificial intelligence to my manage some task. Anything of similar for fusion?

https://www.cgchannel.com/2023/12/how-n ... t-bye-bye/


Inviato dal mio iPad utilizzando Tapatalk

Well, as a regular compositor, and into set extensions and matte painting, I must admit it's very appealing.
At this point, note that the guy used more than copycat to reach his goal ( https://leonardo.ai/ + https://stable-diffusion-art.com/controlnet/ + https://learn.foundry.com/nuke/content/ ... pycat.html ) but yes, very promising...

I'd love BMD surprise us by implementing this kind of tool not years and years after Nuke does. I was yet very pleasantly surprised when after I just tested out some python implementation I found on the web in 2019, BMD realesed their Magic Mask and Depth Map and surface tracker not too late afterwards.

This principle of training your own tools would indeed be cool to perform set extensions and matte painting.

I didn't take time to read this so far but I will
https://www.cgchannel.com/2021/03/nukes ... -networks/
https://learn.foundry.com/nuke/content/ ... pycat.html

Thx for your input anyway

yes, i see that he used many tools, i'm interest in the idea to train with my stuff and optimize the result
like from many times i asked to topaz for videoenhancer to have the ability to feed VeIA with newer video about place, people, or high res photo to help reconstruction of AI on old videos.

Tell me about it...
My last projects all included modified environments (buildings, nature,...) and modestly of course I'm starting to be asked a lot for these skills (CGI, composited set extensions aso...), for shorts, ads, ..., and honestly, stable diffusion seems veeeeerry promising

That copycat feature of Nuke is pretty cool. I might try it out for myself to see if it actually speeds up my workflow for VFX. It's always hard to tell with these types of reviews. I haven't seen anything quite like it in Fusion, but maybe a similar plugin will come out? It's really cool seeing more AI tools coming out. I came across some more that are design focused here: https://appsalon.com.au/artificial-intelligence-design/

The most crucial aspect of copycat is the custom training. It is not just for applying a pre-trained model, you literally train on your own custom data. Obviously you can only train the provided image to image transfer models, but for the specific tasks they are designed for they work nicely. Narrowing the scope and generalization of the model is what is the essence of copycat. It does not try to provide a general model that works on any input, instead you make the model fit your own custom data and how it behaves outside that subset is irrelevant.

This sounds like a variation of the FaceSwap Tech? I used this on a project a few years ago and it takes effectively takes a video input and a series of target images to then generate an overlay image on the source of the target that matches the sources animation. Back then though it was very slow, you would train the model overnight.

This sounds like a variation of the FaceSwap Tech?

It's a variation in the sense that you have the means to train a provided model that is meant for doing something-something, yes. It provides the training facilities directly within comp environment, with all the benefits it brings like traning data augmentation is very easy, adding new training data basically on the fly (you can quickly decide which parts of shot need additional ground truth examples and add them to training data right away etc).

Training speed depends directly on the size of training data set (including resolution), so depending on what you train it for, and how you limit the size, it can reach practically usable results in an hour or faster too.

That sounds very similar, previously I remember the difficulties with the system was setting parameters for training which was a bit of crap shoot as it was very easy to break it and you didn't know until you'd run it if it would work. This was a long time ago and this AI tech seems to improve very quickly. Magic Mask has been huge for my workflow, face swap is not something I'd use often but if this system can be used to retarget deforming patches in shots (e.g. clothing, skin surfaces ...) that would be very useful.

There are different ways to do face swap, you might be thinking of something that is not direct image-to-image transfer. Copycat needs direct spatial alignment of input and ground truth. For fixing deforming surfaces, reflections etc it can work very well though, as ground truth can be produced directly from original by paint work. Training set augmentation with synthetic samples is usually helpful, but it is easy to do by applying randomized transforms which scale and rotate the image.

There was a webinar a few years ago I took part in that might give some ideas for how image to image model can be used (including some funky use cases):