And what about passive effects? Defensive and/or Deflect on the second weapon would work normally even if I didn't attack with it first?
As neither Defensive nor Deflect contribute in any way to their wielder's attack roll, I see no reason they would not apply. Only modifications on the secondary weapon that affect attack dice pools are ignored when using Two Weapon Fighting, as I understand it.