Loyalty is gained and removed with diplomatic mission cards. Units do not have to be on the planet to keep loyalty unless it is subjugated. Then only one unit must be committed to the planet surface to keep subjugation. Loyalty is gained on a system loyal to the opponent with one loyalty token for everyone of that the opponent has placed in that system, plus one additional to make it loyal to you. So that they are canceled one-for-one to neutralize and then an additional loyalty marker would be placed to gain control of that system.
If this is correct, I ask that this or a more organized easy to understand explanation about loyalty winning and subjugation be added to the official rules thread. During our first couple of play through we spent way to much time and effort trying to find simple explanations about this across several rule books in multiple places of each book.