I don't know what your set up is, but if the masks are truly overlapping, you should not be seeing that, which only happens when you have masks close to each other and you are dealing with semi transparent pixel data, or if you have mismatch between alpha multiplication and divide somewhere in your comp.
I can reproduce it by simply putting one mask to the edge of the other, and make it go away by actually overlapping mask.
- sshot-952.jpg (4.08 KiB) Viewed 436 times
- sshot-953.jpg (4.48 KiB) Viewed 436 times
Or you can reproduce it by adding alpha divide node.
- sshot-954.jpg (121.13 KiB) Viewed 436 times
So you are probably dealing with mismatched multiplication operations somewhere in your comp.
Transparency and Color - Alpha Multiplication in Resolve/FusionUnderstanding Premultiply, Unpremultiply and Alpha Divide ( Fusion 8 )