Each additional core set has diminishing returns. The second core set gives you enough class cards to play 2 of each if you don't double up on classes and gives you enough of all neutral cards for 2 players (though you already will have enough of some neutrals with a single core set, such as flashlight). The third core set gives you enough neutrals for 3 players and let's you double up on classes to a limited degree for 2 players. The fourth core set allows you to double up on classes and guarantee you'll have enough core cards for 2 of each for each player. The downside to this plan is that expansions will still only give you 2 of each class card, so you'd have to buy 2 of each of them if you were really determined to double up on classes.
I'd recommend just getting 2 cores and dealing with the small restraints on your selections.
You only use a single set of the encounter cards, regardless of how many sets you own.