So I just recently bought IA and I would like to play it more but I am thinking the only times I can really play it are at tournaments. How competitive will I be with just the base set?

Not very unfortunately. Imperial just aren't strong enough with jabbas realm ( terro and jet trooper). Rebels don't get enough stuff personally as they have no units. You get decent command cards though

You will definitely have playable lists with just the core set, but not really competitive. Imperials are the easiest faction to play with just the base set because elite stormtroopers are pretty good, but Imperials are in a hard spot right now due the recent rule change that causes you to earn points for every figure killed, not just when all figures in a deployment group are killed.

If I were you I would pick the faction that you are most interested in and buy the packs that you like the most within that faction to be able to field a competitive list. If you don't want to buy a lot of packs right away, Scum is probably the easiest way to go because you can field a fairly competitive Scum list straight out of just the core set and Jabba's Realm.