I'll grant you that accidental shuffling of cards from both players can be a problem. But it will be a concern regardless of who controls attachments if there are effects that change who controls a card. And such effects existed in L5R CCG.