Patch Replacer or Object Removal with Transform?

Happy new Year!

I have a project where I want to place three images across the width of the screen. Two of the images have objects that need removing within them - one has two pavement signs and the other has overhead power cables. All the images are static with no camera movement and little movement within the frame. I have done some rough work using Patch Replacer which worked well with the signs but was more tricky with the cables. However, I did my work when the image was at full size and when I use Transform to bring the image to the required size it messes up my Patch Replacer positioning.

Is there a better way to do this? Should I use Object Removal - how does it compare to Patch Replacer? What would be the best workflow?

Thank you

You're doing the removal in Fusion and the Transforms in Edit?

You're doing the removal in Fusion and the Transforms in Edit?

I am doing the removal in the Color Correction tab and Transforms in Edit

Try what I said.

If it were me, I would do the object removal with the material full-frame, render it out as a separate file, then bring each file back in with the object removal already done (basically treating it as a VFX shot). Then once you have the rendered shots, move and place the images and change their positions on the edit page as you see fit. I think this will work better than trying to do everything in one pass.

If it were me, I would do the object removal with the material full-frame, render it out as a separate file, then bring each file back in with the object removal already done (basically treating it as a VFX shot). Then once you have the rendered shots, move and place the images and change their positions on the edit page as you see fit. I think this will work better than trying to do everything in one pass.

Thank you Marc. I had been coming to the exact same conclusion. I have noticed that my object removal/patch replacment work has not been reliable when I use it with Transform. It often seems ok full screen and then things get messed up as soon as I use Transform or start rendering out my transformed images. I'll try doing some full-frame work and then render those out.

Try what I said.

Will that make things more reliable?

Well, I've not had any trouble. (Though I haven't specifically tested the Object Removal effect yet.)

Well, I've not had any trouble. (Though I haven't specifically tested the Object Removal effect yet.)

thanks

I have now tried a different strategy - creating the Object Removal fx separately to the Transforms. However I am getting some unreliable behaviour. For some reason Object Removal seems to forget its clean replacement of power cables over sky and it puts in very obvious bands in their place (see grab). It appears to switch to this spontaneously and it can be corrected back to a good removal by re-using 'Build Clean Plate' but it can mess up at just the wrong time i.e on a very lengthy export.

I've also found that on another shot the object removal doesn't take account of changes in lighting throughout the shot. The removal was pretty good at the beginning of the shot (this image will be reduced to around 1/3 size) but by the end the removal was very noticeable (see grabs). Any advice?

I have also attached grabs (in separate posts) showing my settings for each shot.

I've just seen the definition of 'Assume No Motion' means that the background scene is not changed - does that mean it's a still? Changed in any way or just no obvious movement or camera moves? I had thought the latter but there are luminance changes and some small movements within the shot (rustling leaves) etc. What i am seeing is that I can build a clean plate and it works on a particular section but wehn I move further into the shot it no longer matches. The camera hasn't moved but the light has changed.

I also can't see any difference between the 'Adaptive Blend' and 'Linear' settings.

Screenshot 2021-01-05 at 13.28.33.jpg (97.46 KiB) Viewed 4492 times

Screenshot 2021-01-05 at 13.28.39.jpg (111.51 KiB) Viewed 4491 times

Screenshot 2021-01-05 at 13.29.14.jpg (41.17 KiB) Viewed 4490 times

These are actually the settings for both shots...

Screenshot 2021-01-05 at 14.01.01.jpg (73.41 KiB) Viewed 4489 times

Does anyone have any thoughts on why I am getting these problems and how I can find a way round them.

Thanks

Can't tell about reason for problems, but my suggestion for shots like this would be manual paint, not relying on some magic fill. You can't control object removal one bit, at best you can create clean frames with it that you should then frame-freeze and use for patching the actual footage by tracking-merging-warping-grading-whatever necessary. In current state magic fill methods for moving footage are hit and miss at best, and usually it is miss.

Can't tell about reason for problems, but my suggestion for shots like this would be manual paint, not relying on some magic fill. You can't control object removal one bit, at best you can create clean frames with it that you should then frame-freeze and use for patching the actual footage by tracking-merging-warping-grading-whatever necessary. In current state magic fill methods for moving footage are hit and miss at best, and usually it is miss.

Thank you. When you say 'shots like this' I had hoped that these would be pretty easy shots to process since the camera is fixed and there is little or no movement within the frame. The object removal does seem to work on both the shots but it seems unreliable and on one shot it doesn't take account of gradual changes in lighting and in the other it seems to glitch between a good removal and then replacing the object with a bad match - that can be fixed by asking it to check again but doesn't hold reliably. Is Object Removal known to be unreliable in Resolve?

For manual paint, are you suggesting frame by frame? That's a hell of a task. Can I do manual paints within Resolve?

Thank you. When you say 'shots like this' I had hoped that these would be pretty easy shots to process since the camera is fixed and there is little or no movement within the frame.

I mean for simple shots maybe it is useful for creating clean patches, I wouldn't even try on anything more complicated.

For manual paint, are you suggesting frame by frame? That's a hell of a task. Can I do manual paints within Resolve?

No, frame by frame paint is hard and needs a lot of practice. I mean create a clean patch by cloning-painting-stitching up different frames, freeze that and use as a patch. What I personally like to do if possible is to do live clone, meaning I keep all strokes alive during the whole shot and track them in. This way you get all light changes for free, but it gets more complicated if areas from which you clone are occluded or change in different ways from area you are patching up. You can do it in Fusion too, but I'm not knowledgeable enough to guide you, I use other software myself.

In general, removal goes something like this:
- denoise the plate;
- create patches to cover the area you want to to clean (there are a lot of ways to approach this);
- track those patches in;
- grade to match;
- restore grain on patch areas, keep original grain on areas you didn't touch.

Object removal helps you at best in the patch creation phase, but since it is a black box and you have no control over it, results can work, or not, so you are succumbing yourself into "maybe it works" land which isn't the most productive. It will improve but you can't use tools that are available two years from now, you can only use tools that are here now.

You can do it in Fusion too, but I'm not knowledgeable enough to guide you, I use other software myself.

In general, removal goes something like this:
- denoise the plate;
- create patches to cover the area you want to to clean (there are a lot of ways to approach this);
- track those patches in;
- grade to match;
- restore grain on patch areas, keep original grain on areas you didn't touch.

I've done a little of this, and Fusion can do it pretty well. The multi-stroke clone tool has a couple of quirks -- mainly involving how to set and keep the lifetime of your clone paint stroke to be as long as you need it to cover the entire clip -- but the strokes will stay "live" as you say. Like you say, that can be good since it can add realistic change in terms of lighting, etc, and it an be bad if you're cloning from an area that a bird flies across.

The upside of a manual approach is that you control everything and if you have enough skill you can make something happen rather than just hope it happens. The downside is that it really does require a couple of additional skillsets: cloning, tracking/stabilizing, and Fusion compositing. I.e. you're going to be using multiple Fusion nodes to do something like: a) stabilize the shot, b) clone on the stabilized shot, c) destabilize the shot, and if the shot is moving enough that the stabilize-destabilize results in distortions you may need to track a mask and composite in the cloned part. Or you'll not use stabilize-destabilize but will need to use essentially expressions to apply tracking directly to cloning.

Does anyone have any thoughts on why I am getting these problems and how I can find a way round them.

Thanks

Did you try without Assume No Motion?

Did you try outputting a non-transformed cleaned up version, then importing and doing transforms later?

Both of those suggestions seem reasonable. There is a fairly complex order in which the various parts of Resolve act on a clip (Edit, Edit-applied effects, Fusion, Color, etc) and within each tool there is an order as well. So it's painful, but not totally unexpected that if you apply an effect and a transform things might not work as expected. (Just as, for example, if you try applying a chroma key to a clip and then a black-n-white effect, it will work totally differently from applying a black-n-white effect and then attempting to do a chroma key.)

Hence Marc's wise suggestion.

There are also multiple tools for replacing things, if you have Studio. Patch Replacer is one. Object Removal is another. And using Color and Node Sizing is another (which doesn't require Studio). I think Patch Replacer happens in Edit or Fusion? And Object Removal and the Node Sizing trick happens in Color, I think. Which could interact differently with transforms to the clip.

For something as simple as what it sounds like you're doing, using Node Sizing and Power Windows (with nice feathering at the edges) might work for you.

The multi-stroke clone tool has a couple of quirks -- mainly involving how to set and keep the lifetime of your clone paint stroke to be as long as you need it to cover the entire clip -- but the strokes will stay "live" as you say.

I'm no expert but from what I've seen this is a major pita in Fusion. For example, how do you change the start and end frames of your paint stroke after you have created it?

c) destabilize the shot, and if the shot is moving enough that the stabilize-destabilize results in distortions you may need to track a mask and composite in the cloned part.

If this round trip results in distortions (not just sampling related issues), reverse transform is erroneous and isn't actually mathematical reverse.

The multi-stroke clone tool has a couple of quirks -- mainly involving how to set and keep the lifetime of your clone paint stroke to be as long as you need it to cover the entire clip -- but the strokes will stay "live" as you say.

I'm no expert but from what I've seen this is a major pita in Fusion. For example, how do you change the start and end frames of your paint stroke after you have created it?

Different strokes for different objectives. (Almost get to reuse that old saying.) If you're using the multi-stroke clone brush, you cannot change anything about it. That's part of it's use case: you're potentially doing a lot of strokes and keeping them all as editable splines would bog things down. You can use one of the drawing strokes -- which does remain editable -- but you have to set it to cloning mode manually.

It could be more user-friendly. And I don't usually have many strokes so I should really use the spline mode rather than multi-stroke clone. I just haven't really dug into the other options much and I know the quirk of having to set brush duration once-for-all-time before the first stroke, so that's what I use.

c) destabilize the shot, and if the shot is moving enough that the stabilize-destabilize results in distortions you may need to track a mask and composite in the cloned part.

If this round trip results in distortions, reverse transform is erroneous and isn't actually mathematical reverse.

I may be mis-describing what goes on. The manual has a long example where they show how to paint out a blemish on an actor's head using the planar tracker. (I was assuming the planar tracker here.) And due to the fact that the forehead isn't actually planar and the fact that it moves a fair amount, it does introduce distortions at the edge of the frame that aren't totally reversed by the destabilization, so in the end -- having painted on a stabilized part of the forehead -- they mask their result down to part of the forehead and track it to the forehead. (The stabilize-clone-destabilize creates the appropriate perspective and avoids the clone targets shifting, the mask and track puts that in the right location on the screen.)

I think the issue is that if you do two transforms back to back, Fusion can basically apply them in a single step. It doesn't need to render the first transform to pixels, then apply the second. So it is always reversible, mathematically. But I think if you do something between the two transforms, though, it will have to render pixels and in that case, you could compress large chunks of the original down to a couple of pixels and you can't magically turn that back into a full-resolution original by reversing the transform.

But I'm just sort of repeating something I don't totally understand. My uses of the (planar tracker) have been in cases where destabilizing results in a perfect image and I've never had to do more.

Different strokes for different objectives. (Almost get to reuse that old saying.) If you're using the multi-stroke clone brush, you cannot change anything about it. That's part of it's use case: you're potentially doing a lot of strokes and keeping them all as editable splines would bog things down. You can use one of the drawing strokes -- which does remain editable -- but you have to set it to cloning mode manually.

It could be more user-friendly. And I don't usually have many strokes so I should really use the spline mode rather than multi-stroke clone. I just haven't really dug into the other options much and I know the quirk of having to set brush duration once-for-all-time before the first stroke, so that's what I use.

This I understand, but doesn't multistroke exhibit some properties that drawing stroke doesn't and vice versa? Lets for example take a very frequent use case: I want a clone stroke that: a) is animatable, meaning I can change strokes shape; b) is trackable, meaning I can transform it (and a bunch of other strokes) with tracker; c) length must be changeable, or at least opacity animatable to make it invisible; d) clone transform must be separately controllable (where source comes from).

I think the issue is that if you do two transforms back to back, Fusion can basically apply them in a single step. It doesn't need to render the first transform to pixels, then apply the second. So it is always reversible, mathematically. But I think if you do something between the two transforms, though, it will have to render pixels and in that case, you could compress large chunks of the original down to a couple of pixels and you can't magically turn that back into a full-resolution original by reversing the transform.

This is a resampling problem, there is no real way around it. Ideally you want only your strokes going through stabilize-reverse and then merged over original. Better yet is to do live clone on source but with perspective changes it is a pita. If there is a resampling issue one option is to scale up on stabilize phase, then you at least don't lose stuff due to resolution issues.

Different strokes for different objectives. (Almost get to reuse that old saying.) If you're using the multi-stroke clone brush, you cannot change anything about it. That's part of it's use case: you're potentially doing a lot of strokes and keeping them all as editable splines would bog things down. You can use one of the drawing strokes -- which does remain editable -- but you have to set it to cloning mode manually.

It could be more user-friendly. And I don't usually have many strokes so I should really use the spline mode rather than multi-stroke clone. I just haven't really dug into the other options much and I know the quirk of having to set brush duration once-for-all-time before the first stroke, so that's what I use.

This I understand, but doesn't multistroke exhibit some properties that drawing stroke doesn't and vice versa? Lets for example take a very frequent use case: I want a clone stroke that: a) is animatable, meaning I can change strokes shape; b) is trackable, meaning I can transform it (and a bunch of other strokes) with tracker; c) length must be changeable, or at least opacity animatable to make it invisible; d) clone transform must be separately controllable (where source comes from).

Yes, exactly. There are non-spline paint strokes that are less computationally-intensive, and there are spline-based strokes/shapes that are more computationally-intensive but that retain editable properties. (To be honest, I don't know if modifying the start/end is one of them. I know the shape is modifiable and animatable, but I don't use them -- though I should -- so can't speak authoritatively.

I think the issue is that if you do two transforms back to back, Fusion can basically apply them in a single step. It doesn't need to render the first transform to pixels, then apply the second. So it is always reversible, mathematically. But I think if you do something between the two transforms, though, it will have to render pixels and in that case, you could compress large chunks of the original down to a couple of pixels and you can't magically turn that back into a full-resolution original by reversing the transform.

This is a resampling problem, there is no real way around it. Ideally you want only your strokes going through stabilize-reverse and then merged over original. Better yet is to do live clone on source but with perspective changes it is a pita. If there is a resampling issue one option is to scale up on stabilize phase, then you at least don't lose stuff due to resolution issues.

Again, I'm saying things based on the tutorial, which sort of shocked me when they had to to the final tracking. I'm describing what I perceive to be the issue, which may be misleading. Somehow that final step after destabilize is necessary, but don't take my word on why.

Using trackers to directly affect things rather than stabilize-destabilize is how I originally approached things. That's the way other software I've used works. But it also has various aspects you have to understand -- like which aspect of your tracker do you apply to which aspect of your brushes -- and you have to apply it perhaps multiple times if you're doing multiple things. So the stabilize-do_lots_o_work-destabilize is less work for me when I want to clone out two things, overlay something else, etc, etc, and I'm not having to apply tracking information to a lot of disparate nodes.

Thank you everyone - I really appreciate your advice. Just to clarify, my latest tests have been with ungraded full size shots - I am trying one thing at a time. I'll do my transforms later. Both shots have locked off cameras and very little or no movement within them. The object removal on one shot did work for a while but then lost it as the lighting gradually changed. I had 'Assume No Motion' checked but now I have unchecked it and I am currently doing a scene analysis to see if that helps. I am a bit out of my depth with the other proposed approaches but I'll get learning. I had hoped that Object Removal would do the job well but it is less reliable than I thought it would be.