I don't know if the other parts hold that pin in tension but with the extras off you can get a block of wood under the opposite side of where you are hammering. Make sure you use a punch that isn't tapered and is the right size. Make sure you're not using a tiny hammer either, you want something like a heavy mallet to get things moving.
Some people drill a hole in the wood to catch the pin. If it's soft wood though I just let the pin go into the wood and back it out after.
Airsoft, where nothing is hurt but feelings.