How to "limit" an effect, so it seems like it's "coming" out

I'm doing a more silly porject for my cousin, which includes his pet running out of the burrow, then something exploding from it, with only part of the explosion coming out of it. And I ask, what should I use to accomplish such a task? A mask would do the trick, or maybe something else?

It depends on the footage (is it moving?), how real you want it to look etc. If there's a fixed camera (no movement), it will be fairly easy: Create an explosion (either via the particle system or some stock footage, or a combination), composite it and then indeed mask out the bits that should be hidden by the hole. If you have a bit of movement, you can still do the above, but then also make use of a tracker (like a point tracker) to track the movement and then ensure that the "explosion" and the associated mask will follow the "hole".

If you really want to go the whole way, then you can use a 3D camera tracker.

cheers, Okke