It would put you at a disadvantage when you needed to stay in stealth and you didn't get the DoS needed. It's all about application of skills in the game.
Point is, that if you are spending buckloads of XP into an advancement then you might as well get the comfort of not sucking with the Skill compared to others. Or just suck overall, because you are wasting XP like crazy while the rest of the party just specialize and get much better value out of their hardly earned XP.