Are you allowed to ditch restraining objects (and followers) any time?
For example the Amulet just a second before you want to cast a spell? Or that Clashman who gives you 2 craft penalty just when you are about to engage psychic combat?
Can you do that even when it is not your turn? (You are attacked; or there is a spell cast on you).