If i understand the VoI-rulesheet correctly you do not have to refresh the time counters after the last counter is removed .. except the effect instructs you to do so?

Time X is a new keyword that represents the urgency of the heroes’ quest. When a card with the Time X keyword is revealed, the players put X resource tokens on that card. These tokens are called “time counters.” At the end of each refresh phase, remove 1 time counter from each card with the Time X keyword, if able. When the last time counter is removed, there will be a triggered effect that resolves on that card. Some encounter cards will also remove time counters, making it more difficult for the players to predict when they will run out of time

Correct. Most time effects do instruct you to refresh the time counters when you run out, but there are exceptions - the various locations in The Antlered Crown have time effects which do not refresh their counters when they run out.

Holy carp really? I definitely played that wrong.