If a build with Mux and Palob is facing off with a build with Kyle Katarn, whose ability goes off first? Is it determined by initiative?

Yes it is determined by Initiative. The whole purpose of Initiative within X-Wing is to determine the order for timing conflicts. The player with Initiative resolves any abilities that trigger at the same time as an opponent. So if Palob/Mux had initiative, they would go before Kyle. Pilot skill does not come into play with any of their abilities as they all share the same trigger - "At the start of the Combat phase..."

Initiative determines which player resolves their effects first. If a player has multiple effects with the same trigger, that player decides the order to resolve them in.

So if, for example, you have Kyle Katarn and I have Kaato Leachos and Palob Galodhi, you go first and pass a focus from Kyle to one of your ships. I then choose to activate Palob, and take that focus to put on him. I then use Leachos to take the focus (that I just stole from you) from Palob and put it on Leachos.