Jump to: Board index » General » Fusion

[pretty advanced] Thoughts on (python) AI Automatic Roto

Learn about 3D compositing, animation, broadcast design and VFX workflows.
  • Author
  • Message
Offline

Sam Steti

  • Posts: 2509
  • Joined: Tue Jun 17, 2014 7:29 am
  • Location: France

[pretty advanced] Thoughts on (python) AI Automatic Roto

PostWed Dec 04, 2019 12:38 pm

Hey guys,

For those who were not aware of the possibility of calling python to dive into AI to automatically roto an image or sequence, watch Damian writing about it here, because this is what I'm about to write about here too :
https://www.provideocoalition.com/autom ... gfor-free/

Actually I finally managed to have it work (with pain first because of OSX ".DS_Store" interfering) for sequences and I'm now really happy to be able to rely on it for special occasions...
The process is long - python perf limitation for now - and connected to the final file size, so I decided to make pics at 270 px height, to finally use x4 superscale in Resolve to rebuild a matte sequence. Here is a example :
902_0010 160_matte.jpg
902_0010 160_matte.jpg (4.54 KiB) Viewed 1520 times


Now, the real question : among soooo many options to use mattes in Fusion (yes, using Fusion exclusively is my goal), what would you choose to result in a perfect keyed talent, with as few nodes as possible ? Hmmm ?
I have the raw whole sequence + the whole matte sequence (clip or still sequence)

I'd personally opt for using channel booleans, edge matte refining and matte control but what d'you think ?
*MacMini M1 16 Go - Ext nvme SSDs on TB3 - 14 To HD in 2 x 4 disks USB3 towers
*Legacy MacPro 8core Xeons, 32 Go ram, 2 x gtx 980 ti, 3SSDs including RAID
*Resolve Studio everywhere, Fusion Studio too
*https://www.buymeacoffee.com/videorhin
Offline
User avatar

Bryan Ray

  • Posts: 2491
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: [pretty advanced] Thoughts on (python) AI Automatic Roto

PostWed Dec 04, 2019 4:47 pm

Handling the matte isn't going to be any different than if you received it from an outsourced rotoscoper.

You can skip the ChannelBooleans. MatteControl has that built in. Just put the matte into the green input and set Matte Combine to Combine Alpha, or Combine Red or whichever channel if your auto-rotoscoper spit out a matte without an alpha channel. Then tick the Post-Multiply switch to associate.

After that, it will work just as though you were feeding it a key.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

Sam Steti

  • Posts: 2509
  • Joined: Tue Jun 17, 2014 7:29 am
  • Location: France

Re: [pretty advanced] Thoughts on (python) AI Automatic Roto

PostThu Dec 05, 2019 8:56 am

Hey Bryan, glad you popped,in ;)
Bryan Ray wrote:Handling the matte isn't going to be any different than if you received it from an outsourced rotoscoper.
Well, this I know; however please note that so far the matte produced this way is said being particularly "rough", therefore not really feathered etc... It was included in my question actually, though anyway I'll apply what I usually do when doing edge matte/core matte technique...
You can skip the ChannelBooleans.
Hmmm, actually I don't know a lot about this node since I never use it, except it would probably be super handy in many situations... The problem is I don't figure out the situations when it should be used :D . I thought it could be there... See below why...
MatteControl has that built in. Just put the matte into the green input and set Matte Combine to Combine Alpha, or Combine Red or whichever channel if your auto-rotoscoper spit out a matte without an alpha channel. Then tick the Post-Multiply switch to associate.
Well Matte Control is a node I even know well; I think it's the node used in ANY of my project, but more fore gamma, erode/dilate, spill control, blur etc. BTW there's indeed no alpha channel in the resulting matte.

I realize I have to change my POV on the way I handle these nodes (for I always use them the same way).
Here is what I'm ashamed of : the matte will probably always look like the one I posted above, so I don't figure out how a rotoed people can be made out by selecting "whichever channel" as you wrote. I haven't been used to select random options... Should I select red because the matte pic is "reddish" ?
What's the influence of the color of the matte here ? (you see my confusion ? :lol: )
And that's also the reason why I wondered wether I may use Channel Booleans to do "to alpha -> red" or not
*MacMini M1 16 Go - Ext nvme SSDs on TB3 - 14 To HD in 2 x 4 disks USB3 towers
*Legacy MacPro 8core Xeons, 32 Go ram, 2 x gtx 980 ti, 3SSDs including RAID
*Resolve Studio everywhere, Fusion Studio too
*https://www.buymeacoffee.com/videorhin
Offline
User avatar

Bryan Ray

  • Posts: 2491
  • Joined: Mon Nov 28, 2016 5:32 am
  • Location: Los Angeles, CA, USA

Re: [pretty advanced] Thoughts on (python) AI Automatic Roto

PostThu Dec 05, 2019 3:43 pm

The channel selection you do in the Matte Combine control will do exactly the same thing as using the ChannelBooleans for that purpose. It is, in fact, just a mini version of the Boolean built into Matte Control for convenience. The color of the matte doesn't matter at all, except that you'll want to be sure that the value of whatever channel you choose to use is 1 where the matte is supposed to be solid. Since it appears a little red, the red channel is obviously the one to use, but you might need to threshold it with a Bitmap node to ensure the range is correct.


It's a bit outside the scope of the topic at hand, but ChannelBooleans is a general utility node with a lot of uses: Adding CG components together is a common one. Diffuse + Reflect + Specular is a pair of ChannelBooleans in Add mode (but don't forget to set Alpha to Do Nothing). You can use it to pull a Z channel out, manipulate it with a BrightnessContrast, then inject it back into the main image. Use CreateBump to make normals out of your alpha, like this:

Code: Select all
{
   Tools = ordered() {
      Blur1 = Blur {
         Inputs = {
            Filter = Input { Value = FuID { "Fast Gaussian" }, },
            XBlurSize = Input { Value = 25.2, },
            Input = Input {
               SourceOp = "BrightnessContrast1",
               Source = "Output",
            },
         },
         ViewInfo = OperatorInfo { Pos = { 385, -16.5 } },
      },
      ErodeDilate1 = ErodeDilate {
         Inputs = {
            Filter = Input { Value = 2, },
            XAmount = Input { Value = -0.18, },
            Input = Input {
               SourceOp = "Background1",
               Source = "Output",
            },
         },
         ViewInfo = OperatorInfo { Pos = { 385, -82.5 } },
      },
      BrightnessContrast1 = BrightnessContrast {
         Inputs = {
            Alpha = Input { Value = 1, },
            Gamma = Input { Value = 1.95, },
            Input = Input {
               SourceOp = "ErodeDilate1",
               Source = "Output",
            },
         },
         ViewInfo = OperatorInfo { Pos = { 385, -49.5 } },
      },
      CreateBumpMap1 = CreateBumpMap {
         Inputs = {
            FilterSize = Input { Value = FuID { "5" }, },
            Input = Input {
               SourceOp = "Blur1",
               Source = "Output",
            },
            SourceChannel = Input { Value = FuID { "Alpha" }, },
            HeightScale = Input { Value = 100, },
            BumpmapTextureDepth = Input { Value = 3, },
         },
         ViewInfo = OperatorInfo { Pos = { 385, 16.5 } },
      },
      Background1 = Background {
         CtrlWZoom = false,
         Inputs = {
            Width = Input { Value = 1920, },
            Height = Input { Value = 1080, },
            Depth = Input { Value = 3, },
            ["Gamut.SLogVersion"] = Input { Value = FuID { "SLog2" }, },
            TopLeftRed = Input { Value = 0.188, },
            TopLeftGreen = Input { Value = 0.020304, },
            TopLeftBlue = Input { Value = 0.020304, },
            EffectMask = Input {
               SourceOp = "Ellipse1",
               Source = "Mask",
            }
         },
         ViewInfo = OperatorInfo { Pos = { 220, 49.5 } },
      },
      Shader1 = Shader {
         Inputs = {
            Ambient = Input { Value = 1, },
            Diffuse = Input { Value = 0, },
            Specular = Input { Value = 1.449, },
            Reflection = Input { Value = 0, },
            EquatorAngle = Input { Value = -25.4, },
            DiffuseCurve = Input {
               SourceOp = "Shader1DiffuseCurve",
               Source = "Value",
            },
            SpecularCurve = Input {
               SourceOp = "Shader1SpecularCurve",
               Source = "Value",
            },
            SpecularGreen = Input { Value = 0.9, },
            SpecularBlue = Input { Value = 0.9, },
            Input = Input {
               SourceOp = "ChannelBooleans1",
               Source = "Output",
            },
         },
         ViewInfo = OperatorInfo { Pos = { 550, 49.5 } },
      },
      Shader1DiffuseCurve = LUTBezier {
         KeyColorSplines = {
            [0] = {
               [0] = { 0, RH = { 0.333333333333333, 0.333333333333333 }, Flags = { Linear = true } },
               [1] = { 1, LH = { 0.666666666666667, 0.666666666666667 }, Flags = { Linear = true } }
            }
         },
         SplineColor = { Red = 255, Green = 128, Blue = 0 },
         NameSet = true,
      },
      Shader1SpecularCurve = LUTBezier {
         KeyColorSplines = {
            [0] = {
               [0] = { 0, RH = { 0.333333333333333, 0.333333333333333 }, Flags = { Linear = true } },
               [1] = { 1, LH = { 0.666666666666667, 0.666666666666667 }, Flags = { Linear = true } }
            }
         },
         SplineColor = { Red = 255, Green = 255, Blue = 255 },
         NameSet = true,
      },
      ChannelBooleans1 = ChannelBoolean {
         Inputs = {
            ToRed = Input { Value = 4, },
            ToGreen = Input { Value = 4, },
            ToBlue = Input { Value = 4, },
            ToAlpha = Input { Value = 4, },
            EnableExtraChannels = Input { Value = 1, },
            ToXNormal = Input { Value = 0, },
            ToYNormal = Input { Value = 1, },
            ToZNormal = Input { Value = 2, },
            Background = Input {
               SourceOp = "Background1",
               Source = "Output",
            },
            Foreground = Input {
               SourceOp = "CreateBumpMap1",
               Source = "Output",
            },
         },
         ViewInfo = OperatorInfo { Pos = { 385, 49.5 } },
      },
      Ellipse1 = EllipseMask {
         Inputs = {
            Filter = Input { Value = FuID { "Fast Gaussian" }, },
            MaskWidth = Input { Value = 1920, },
            MaskHeight = Input { Value = 1080, },
            PixelAspect = Input { Value = { 1, 1 }, },
            ClippingMode = Input { Value = FuID { "None" }, },
            Width = Input { Value = 0.247398607384522, },
            Height = Input { Value = 0.247398607384522, },
         },
         ViewInfo = OperatorInfo { Pos = { 110, 49.5 } },
      }
   }
}


Get the intersection of two mattes with And or Multiply mode. Lots of possibilities; way more than I could possibly list.

Back on topic...

Any matte manipulation you need to do prior to injecting it would be done in the branch before it's put into the MatteControl.

As of right now, I haven't yet heard of an AI-driven rotoscoper that produces production-quality work. Rotobot from Kognat apparently does a good enough job for temps, although I couldn't get the plug-in to run when I tried it at work, and his web store wouldn't let me download a second copy for experimenting at home, so I didn't really get the chance to try it for myself. That's the best one I've seen in demos so far. What that means is that if you're using the auto-roto for core and garbage mattes, you'll probably be doing some work prior to the matte control to create an acceptable composite matte. Here's where you'll likely really use ChannelBooleans.

Let's suppose you have sufficient control to get your auto-roto to give you not only a core matte but also a garbage matte. Both of these mattes are white where the actor is and black outside. You're going to do some keying and manual roto to get a quality edge, which will be white along the edges but may be any color elsewhere.

Combining the core and edge mattes is easy. ChannelBooleans has a Maximum mode, so wherever the edge matte drops below white in the core, the automated core matte will fill it in.

There may still be non-black areas outside the talent, so you need to use the garbage matte to knock those out. There are a couple of options. You could use a ChannelBooleans in Negative mode to invert the garbage matte, then a second Booleans to either Clear or Subtract from the main matte. Or you could use a single ChannelBooleans in Multiply or And* mode to intersect the two mattes—any pixel that is black in either image becomes black in the output. That would complete your matte, which you'd then inject with MatteControl.


*AND is, I believe, mathematically identical to Multiply, but some people find it easier to understand semantically: I want white where this AND that are both white.
Bryan Ray
http://www.bryanray.name
http://www.sidefx.com
Offline

Sam Steti

  • Posts: 2509
  • Joined: Tue Jun 17, 2014 7:29 am
  • Location: France

Re: [pretty advanced] Thoughts on (python) AI Automatic Roto

PostThu Dec 05, 2019 4:30 pm

HAHA Bryan, that's a super interesting topic, though I admit I put 2 topics in one. Thx to answer here to discuss it. I'll split my post into 2 separate parts for more convenience.
And I don't have enough time right now but I can tell right now that I just understood that :
The color of the matte doesn't matter at all, except that you'll want to be sure that the value of whatever channel you choose to use is 1 where the matte is supposed to be solid. Since it appears a little red, the red channel is obviously the one to use, but you might need to threshold it with a Bitmap node to ensure the range is correct.

Another quick word at once about channelBooleans : I had hard times to understand and apply a technique for light wrapping, using channelBooleans and "Exlusive Or" inside it, to target the edges only of a matte (more or less depending on the pre-blur power on it too).
That understood, I found it so efficient that I had to find more useful situations, especially with mattes which I'm dealing with more and more these times. So I'l carefully read your post, check your code I couldn't now and ...come back soon :ugeek:
THx again
*MacMini M1 16 Go - Ext nvme SSDs on TB3 - 14 To HD in 2 x 4 disks USB3 towers
*Legacy MacPro 8core Xeons, 32 Go ram, 2 x gtx 980 ti, 3SSDs including RAID
*Resolve Studio everywhere, Fusion Studio too
*https://www.buymeacoffee.com/videorhin

Return to Fusion

Who is online

Users browsing this forum: No registered users and 47 guests