Page 137 says that a minions loyalty is always equal to your Fellowship.
A character sheet I've downloaded seems to also add the number of talents to the loyalty score.
Is there any other way to increase loyalty? Forcing every single PC to buy up a lot of fellowship just to control their minions seems a tad silly.