I was playing as the Dervish.... Sam ... joined as an ally, then I said I am using my power to switch hands between the three of us involved in the encounter(Myself, defense and Sam). As soon as I indicated this he played a Force Field card to stop his alliance and withdraw himself.
That isn't a legal play. Since you have already announced the use of your power, the only actions that can be played now are actions that cancel or otherwise modify your power use. So he could play Cosmic Zap, or something that says "if a player is about to take cards from you," or something else specifically related to your power. But his Force Field is not a specific response and thus has to wait for the current action to finish.
In other words, Cosmic Encounter does not have an "action stack" like, say, Magic the Gathering. Responses to actions (and responses to responses) of course can stack up, but "top-level" game actions have to wait their turn.