The question I have on this topic is if you have already spent XP on a non-class skill... say you are a Bounty Hunter Assassin and bought 2 ranks in Ranged (Light) but then get Gageteer as a new specialization. do I get the 10 extra XP I spent on Ranged (light) back?
That was answered "No" in an episode of Order 66.
So it becomes a penalty to take a specialization after taking some of it's class skills? that doesn't seem right.
There's no penalty involved there. It costs a certain amount to buy into your Nth specialization. That amount never changes. Imagine the paperwork involved in tracking how much you spent for each rank of every skill, and determining how much each specialization would 'refund'. Not a big deal if you buy into the spec early, but what if you're 400 XP into a game. Do you just suddenly get 3 new specializations for 'free' because you've bought some skill upgrades in the past?
You gained something for the skill points you spent. Improved capability in a non-career skill. The fact that you've just made it a career skill doesn't mean you didn't get the benefit of having those ranks *before* they were career skills. Had you bought into the spec *before* putting the ranks into the skills, you wouldn't have been able to make use of that improved capability yet.