My game "group" consists of solely me and my brother, and we each play 2 investigators. We play with all expansions we have, and we do not remove any cards: Curse of the Dark Pharaoh, Black Goat of the Woods, Dunwich Horror and Kingsport Horror. When I said we do not remove any cards I meant just that- we don't even remove allies. Instead, we remove 4 allies the first time the terror level rises and 3 allies on the following times (there are 34 allies, so this methods makes them equilivant to 11). However to balance this, when we shop for allies at Ma's, we only draw 2 allies and choose one. I found out that it's much funner to choose between just 2 allies rather than them all (even if it's just 10 or so).
What about you? How many players are you? how many investigators? and which expansions?