Need help on rotoscoping.

I need help on trying to make the area inside the polygon get cropped out rather than the area outside the polygon, any button or trick on how to do this?

You just wire two polygons after each other and set it to subtract.

I'm a little confused. There's an Invert checkbox on all the masks that I'm aware of, including Polygon and Bspline, the two you're most likely to use for roto work. Is this not what you want?

If you're using multiple masks and routing them through each other and you want one to punch a hole in the other, as UserNoah says you'll switch the Paint Node of the second one to Subtract.