I tend to use Adversary sparingly, depending on who the NPC is and how they fit in with the story. For example, my players encountered a smuggler opponent (Rival) who usually has the Adversary 1 Talent, as this smuggler was a relatively small part of the adventure, I ignored Adversary for that encounter.
However, in as far as upgrading the difficulty, I have introduced 'qualities' for certain situations. One being the addition of 'Difficult 1' quality for lightsabers that are difficult to use without training, i.e. You don't have a skill in using a lightsaber this triggers the Difficult 1 quality, the difficulty is upgraded once potentially resulting in a Despair symbol which in this case would cause damage to the user. Another one is 'Potent' for poison/venom that is particularly potent and more likely stay in the system for more than one round (using a Despair symbol in a Resistance check).
Though it could be argued that this is what Destiny Points are for, I didn't want to have to spend a DP for each swing of a lightsaber by an untrained user, or each bite from a creature I created who's venom I decided was particularly nasty.
Question: Does Adversary work in vehicle/space combat? For example if attacking a starship being piloted by a smuggler who you are using the Adversary Talent with, does this upgrade the attack difficulty against their ship?