I just bought the core set with the intention of playing casual fun games with my wife. I am wondering is should I be buying more to really enjoy a good game with her? If I do get more what suggestions are there and why?

Really depends whether you want to do any deck building or not. If you just want to play casual games and don't want to bother with custom decks, a single core set with just the 4 starter decks is a great option. If I were you, I'd play with the starter decks for now and consider buying more cards later when you find that you really love the game and want to start tinkering with decks.

When you do decide to buy more I would recommend a second core set.

Just to add to what dbmeboy had to say... If you find that you really like the game I would recommend that your next purchase be a second core as that will really allow you to build much more consistent decks. Furthermore, if you do play more and find that either of you prefer a specific faction I and the community may be able to provide some advice as to what packs support the specific factions best. Having said all this, I would start off with 2 EoD after a second core as this fleshes out the Smugglers and Scum to a great degree.

'ed by Toqtamish on the second core!

The core set in itself can be a fun, complete game. You'll notice that the stronger affiliations to play with 1 core is Jedi and Sith. With 2 core sets you'll get more consistency in your decks, as you can have 2 copies of most objective sets in your deck. Other than that, the smugglers & spies and the scum & villainy affiliations only have 1 objective set each in the core set. The edge of darkness expansion is basically the smugglers and scum core set. Just like the core set, the edge of darkness expansion only gives you a single copy of most objective sets, so for consistency and competitive options, it's good to get 2 core, 2 edge of darkness. Force packs can be grabbed at your discretion. Balance of the force is mostly for 3-4 player games, there is only 1 objective set for each affiliation in it that can work in 1v1, and those sets are limited to 1 copy per deck.

You probably don't need to hear this again, but I would definitely stick with just the core set until you get some more play with the starter decks. I have been playing this game since its release and even though I love getting new cards and building new decks, I still really enjoy a casual game with the starter decks. Even after all the cards that have expanded this game and all the time I've spent contemplating decks, the starter decks still haven't lost their charm.

Of course, this may be because I have the option of playing other decks besides starters, so when you start feeling saturated with starter decks, I would definitely purchase that second core set because it gives you a great foundation for building lots of different deck types.

If you do decide to expand later, though, it's worth noting that there are still ways of building competitive decks using nothing but 2x Core Sets. For the longest time, the same three Sith objective sets found in the Core Set were considered staples: 2x Fall of the Jedi, Counsel of the Sith, and The Emperor's Web still find their way into a lot of Sith decks. That said, the further you expand into the Force Packs and such, the more options become available, including that of building decks around the "fringe" affiliations: Smugglers and Spies for LS, and Scum and Villainy for DS.