I have enough trouble bumping ships around in short sleeve shirts. It's not like I'm a chunky dude and have big sausage fingers or anything. And I'm not the LEAST coordinated guy on the planet, but you'd have to have rolled REALLY high for your dexterity when making your character to wear a costume and play this game without it looking like some kaiju movie or something.
Then you're doing it wrong. Ideally you want to move your figures around using the Force and keeping your little paws as far away from everything as possible.