I have a group that pretty much plays exclusively 3 or 4 players. The game actually scales pretty well, but it does seem significantly easier as long as you are using custom decks. I'd say it gets easier the more players you have but not so easy that the game isn't fun.
With one card per player being drawn from the encounter deck you can get buried pretty quickly if you don't have the tools available to handle them. For example, a bunch of high threat locations in the staging area and no Northern Trackers in play. A couple of turns like that and you are having to commit a lot to the quest and can get stuck with the next set of big bad guys while everyone is exhausted.
I like the 3 to 4 player games because it can let the players really customize their decks. The way we have ours built allows each person to fill a niche or two that they enjoy playing. I'm pretty excited to play Hunt for Gollum with 4 players because some of the cards in the encounter deck get exponentially worse the more players you have.
