There is absolutely no in-game reason not to take the free 'evade'.
What about being stressed?
On the issue at hand, while you can't expect your opponent to let you do the takeback, if you are the opponent I can't see any reason to refuse beyond being a hardcore Spike.
There are situations where the presence or abscense of an evade token could affect an opponent's targeting priority. You could be in the presence of 2 VT-49s, but shy away from the one carrying the rebel captive and target the one with Ysanne, having forgotten about the evade token (because this would be the first round it came into play, for example). If, halfway through the attack, the player remembers the evade token and wants to make it reappear, what is to say that the opponenet would not have targeted the other VT-49 instead (and taken the stress)?