Jump to: Board index » General » Fusion

Optical flow and frames deformation

Learn about 3D compositing, animation, broadcast design and VFX workflows.
  • Author
  • Message
Offline

Umberto Uderzo

  • Posts: 169
  • Joined: Fri Mar 13, 2015 12:19 am

Optical flow and frames deformation

PostMon May 07, 2018 10:40 am

Would it be possible to track the optical flow for a sequence, then use the optical flow vectors to deform other sequence frames, or better different pictures?
Offline

alan bovine

  • Posts: 117
  • Joined: Mon Aug 07, 2017 6:20 pm

Re: Optical flow and frames deformation

PostMon May 07, 2018 11:21 am

Yes.
Fusion video tutorials : https://www.youtube.com/channel/UCTCeDas53OEcWcRujkQiwLg/videos?view_as=subscriber
Fusion Tools : https://github.com/statixVFX/stx_tools
Nuke 2 Fusion nodes : https://github.com/statixVFX/nuke2fusion
Offline

Umberto Uderzo

  • Posts: 169
  • Joined: Fri Mar 13, 2015 12:19 am

Re: Optical flow and frames deformation

PostMon May 07, 2018 12:47 pm

Got it. Sorry for the dumb question.
Offline

alan bovine

  • Posts: 117
  • Joined: Mon Aug 07, 2017 6:20 pm

Re: Optical flow and frames deformation

PostMon May 07, 2018 7:06 pm

The optical flow nodes compute guestimated trajectories for pixels, its literally a vector map. Shuffle/ChannelBoolean the vectors out of the layer so you can use them on another image. For instance the vector distort, displace and vectorBlur notes can input RGBA vectors (vectorDistort and vectorBlur can also use the channel layer name "Vector")

You can also feed the vectors to a particle system, so you can drive particles with velocities from the image....

Beauty of fusion, just rewire stuff until something magically happens :)
Fusion video tutorials : https://www.youtube.com/channel/UCTCeDas53OEcWcRujkQiwLg/videos?view_as=subscriber
Fusion Tools : https://github.com/statixVFX/stx_tools
Nuke 2 Fusion nodes : https://github.com/statixVFX/nuke2fusion
Offline

Umberto Uderzo

  • Posts: 169
  • Joined: Fri Mar 13, 2015 12:19 am

Re: Optical flow and frames deformation

PostTue May 08, 2018 10:00 am

Thanks Alan,

I was trying to mimick a Superresolution algorithm by tracing the optical flow for each frame then, frame by frame, deforming the previous and the next frame to match the current frame and merging everything toghether at 2x resolution then rescale to original resolution.
Ideally, should get a slight increment in definition but the vector map i'm getting out of the optical flow looks pretty unstable.

There is a tool called "Repair frame" that works in a similar fashion but looks way more stable in tracking.
Don't know what i'm doing bad. :oops:
Offline

alan bovine

  • Posts: 117
  • Joined: Mon Aug 07, 2017 6:20 pm

Re: Optical flow and frames deformation

PostTue May 08, 2018 10:51 am

You're on the right track, I've built a tool that does this, but its not meant for "super resolution" but rather for doing simple denoising. It does an average or a median of 3 frames, I use it a lot for denoising indirect illumination passes in CG. But the theory is the same, you push/distort the previous and next frame into looking like the current frame.

Check out my stx_vectorDenoise tool: https://github.com/statixVFX/stx_tools
Fusion video tutorials : https://www.youtube.com/channel/UCTCeDas53OEcWcRujkQiwLg/videos?view_as=subscriber
Fusion Tools : https://github.com/statixVFX/stx_tools
Nuke 2 Fusion nodes : https://github.com/statixVFX/nuke2fusion
Offline

Umberto Uderzo

  • Posts: 169
  • Joined: Fri Mar 13, 2015 12:19 am

Re: Optical flow and frames deformation

PostWed May 09, 2018 7:24 am

Thank you Alan,
i'm glad to know i'm on the right track, i'll give a read at your page asap!
Cheers!
Offline

nbyteFX

  • Posts: 5
  • Joined: Wed Apr 11, 2018 9:26 am
  • Real Name: Nico Massimo

Re: Optical flow and frames deformation

PostWed May 09, 2018 4:03 pm

@alan bovine: How do you feed the vectors to a particle system, so you can drive particles with velocities from the image?
Can you enlight me!

Thanks,
Nico
Offline

nbyteFX

  • Posts: 5
  • Joined: Wed Apr 11, 2018 9:26 am
  • Real Name: Nico Massimo

Re: Optical flow and frames deformation

PostFri May 11, 2018 4:07 pm

Ok, I found a way to do it.
For those who are interested:
1. Assign a optical flow node to your video footage
2.Copy aux node to move the vectors to color
3. After your particle emitter use a pCustomForce node and pipe the output of the copy aux node to the image1 input of pCustomForce.
4. use a expression in ForceX like: "getg1d(px,py)" -> which means read the green channel of image1 input at position x, y.
5. ForceY: "getr1d(px,py)" - here I use the red channel.

Return to Fusion

Who is online

Users browsing this forum: No registered users and 4 guests