Basically, my question is this: Can you only use Talents that require an Action to activate them during encounters?

If so, what would you do when a player wants to use Encouraging Song (p. 88 of ROT) to heal strain or help in a non-encounter/combat situation?

Encouraging Song - While equipped with a musical instrument, your character may use this talent to make an Average Charm or Verse check. For each success the check generates, one ally within medium range adds boost to their next skill check. For each advantage, one ally benefiting from Encouraging Song heals 1 strain.

You can use talents whenever. The GM should use good judgment if the party just wants to sit around and spam abilities.

"Can we sit and heal all the way back to full, even if it takes 5 attempts each for us to do it?"

"Sure. The village is burning at the moment, but take all the time you need..."

I think the answer to your base question is typically yes, you can allow structured powers to activate in non structured time, but that there needs to be trust between the player and GM not to abuse it. It should also fit narratively.

For example, a character making a skullduggery check to disarm a trap or pick a door, a medicine check to heal somebody, a perception check to search a room, sure fire away with your encouragement to give them boost dice. Trying to make stealth checks, probably not since the song needs to have some element of sound associated with it.

The purpose of Encouraging Song is to assist with a skill check, the strain recovery is a bonus. So that needs to be factored into its use as well. In non structured time you could query the song performer as to what skill check they would like to assist the other player with, and if they can't give a good answer, then no song. If there is no skill check being called for, an thus no benefit to the encouragement from the song, then no song. Keep in mind that the GM calls for checks, the players don't, otherwise you'd have a bunch of lugs performing combat checks towards the ground to generate advantage to heal strain.

If you have a player who just wants to keep spamming their song, making skill checks to generate advantage to regain strain for the party, that's where my quit being a twink reflex tends to flare up. I tend to take the direct approach and either outright say no, or explain my logic about why it's allowable in this special circumstance once, but not allowable as a spam strain recovery device. If they want the strain recovery like that they should use the Heal Spell.

I think probably the best compromise here would be to allow the performer to use the talent and give boost dice (and strain recovery advantage willing) to the strain recovery roll at the end of the encounter. This could narratively fit as the bard is trying to help calm his friends nerves after the fight. This way the performer gets benefit out of their song, but then you can move on without having them spam it over and over.

A common way to deal with this is for GM’s to say “you have time for one action before the story moves forward, wdyd?”