Does fantasy flight use a specific system or algorithm to decide how many squad points a ship is or do they just think of a number that seems fair? I've been wanting to try making my own ships for the game, but I am always worried about making a ship over powered and under priced, so if there is a system i could use, that would definitely be nice

Bit of both, it seems. They do have some sort of system, but ships tend to deviate from it after testing. There's a few fanmade systems around, but it's better to only use that sort of thing as a guideline.

Instead, think about:

100pts of that pilot

100pts of that pilot with the most powerful upgrade they can have (say, HLC Scyks)

100pts with one unique pilot and other generics from the faction

Basically what you want to do is decide which pilot+upgrade combos you want to allow and price the pilots so that you can control how many of that combo can be fielded.

It seems that they used to for the first 3 Waves, but abandoned the practice for Wave 4 onward. And the more recent stuff seems to be more fairly pointed out.