Does the first player token only change hands in an AGE CHANGE PHASE?

Thanks.

That's what it says in the rules. When we play, we pass the first player token at the end of each adventure. Although not entirely necessary, It serves as a reminder of whose turn it is after we complete the bid for Conan.

Well the idea of the token is mostly to keep the track of the player who will act next while there is some kind of pause - like bidding for Conan, Age changing, dinner etc.

Yes, but he has a role to play in the AGE CHANGE PHASE...

We usually pass it to the player who needs to refresh the fate dice pool. The dice that you collect serve as a clear reminder of who goes next.

There is no need for a first player token in my opinion.