Why not post the actual code or screenshot of how it looks. Instead of just nodes with nothing to show how it looks or how it should look. Its hard to visualize what is actually happening... or not happening.
From your node set up it seems polygon is masking the merge operation but I have no way of knowing what the mask actually is or what is supposed to do.
When you apply mask, blue input to merge node, it will mask out the foreground input. Probably in your case it only masks out one of the outlines. To apply it to both you need another node after merge node to apply mask to. For example Matte Control.

- sshot-970.jpg (165.11 KiB) Viewed 596 times

- sshot-971.jpg (162.26 KiB) Viewed 596 times