I used play the SW Card game VERY casually. I bought the core, then the first deluxe, then edge of darkness. I bought those three because I felt like I needed them to have a full set of each faction or a complete game.

Then I really didn't play much, especially after picking up Imperial Assault.

BUT - I'm thinking about getting back into it. So, what would be good to get now? I'd just like to get something that would add to all the factions together, not focus on any single on in particular. Should I get multiple packs?

For casual I would say get Between the Shadows and when Imperial Entanglements is released scoop that up. I would recommend for the packs to do a little research on cardgamedb.com and see if any of the cards from the Force Packs are ones that you really want and just pick up those packs in particular if you want a deeper card pool.