How many turns did you get in over the game time?(And how long was the time limit. What did things look like at the end. We can't make a very informed decision without those, but I would trust your intuition.
I would agree that additional information is needed before you call him "stalling". Some people play at a slower pace, and some people play at lightning pace. If he always takes 3-4 minutes (which is a long time, imo) then that's his pace of play, and you could ask him to pick up the pace. If you notice he's doing it every time he has the point advantage, then I would say he's stalling.
I don't think it should've rattled you to the point where you're making "uncharacteristic" moves. That's like putting a poker player on tilt.
speaking with other players after the tournament, they told me he was quick with his turns