To say that the Tarkin Doctrine (fear) wouldn't work logically is true. But people aren't logical.
There is a reason why a single gunman can hold off an entire crowd of unarmed people. No one wants to be the next person to get shot.
Now, it's possible for people to band together and overcome the gunman. Logically speaking, a single gunman has no chance against a crowd of unarmed people. He will eventually be overcome. But people aren't logical. People are fearful. That is the point of the Tarkin doctrine, and why it could have been so effective. It's the same principle, just on a galactic scale.
Where there exists hope, people can set aside fear. But had the Death Star annihilated the Rebel Alliance, that "New Hope" would have been dashed, and the dark times would have continued unabated. People would be hedging their bets, turning in anyone suspected of Rebel activity just so that they could keep their own family safe.