I play 4 player games occasionally, and my core group is 3 so we run into quite a bit of overlap. Which is why I have 4 cores, in addition to an extra pair of Rite of Seeking cards.

With all of the above we routinely play 3 player with everyone equipped to go all out on hard and expert.

I wouldn't seriously consider playing with more then 1 player without a second core. Extra copies of stuff like Shrivelling and Dr. Milan are vital, and then there's Guts and Overpower, you can't reasonably play combinations like Skidd's + Roland or Wendy + Agnes if you cant supply them with key skill cards. Unless you're willing to settle for standard difficulty play.

My collection might be considered a little extreeme, but having a "complete" set feels good, experiences may vary of course. The only other card I imagine I might still lack is an extra couple copies of Shortcut.