Does adversary upgrade the dice of passive effects like terrifying? If so an inquisitor makes you roll against 3 red dice instead of 3 purple whenever they show up.

No, just combat checks (attack rolls, basically). I'd consider including it in opposed Discipline checks when using Force powers on an NPC with Adversary, especially if it was a Nemesis, but that's just my opinion.