I'm not sure which hard counters you mean. Effects like Proton Bomb, Advanced Homing Missiles, and Wampa that ignore shields hit Phantoms hard, but not really any harder than they hit other ships with 3 Agility.
Most of those other 3 Agility ships don't spend most of their time flying around with a functional 5 Agility though.
And a good chunk of those 3 Agility ships also have a 3 Hull, which prevents them from getting one-shotted by those weapons.
But the phantom also has two shields, which often protect it a bit more against "early" crit effects. For example, while a Decivader still hurts a phantom but its initial attack isn't nearly as likely to finish it as it is an interceptor. Its four health also means that auto-hit effects aren't as likely to kill it (AB, Feedback array, the new Scum droid, etc..) There's some balance there.
