So if my unit has a Stun Token on it, and it moves (3) left dial, charges (starburst symbol) right dial. Can my opponent spend the Stun Token to cancel the modifier dial to make me collide with the enemy unit (and get a Panic Token) and not attack?

Absolutely. Spending the stun token cancels the modifier, so the action is not a charge and colliding with the enemy gives a panic token. That's my understanding.