I've played near every week since wave four using defenders in all but two games and I've never got a defender stressed and I've only lost three defenders to enemy fire, there's no need to ever stress a defender.
Why don't your opponents starting bringing ways to deal stress? Not a knock on them, but this is kind of hard to fathom for me. If I was walking all over my opponents with the same ship, I think they'd respond.