Okay so I just started playing and basically keeping to the manual builds. But can you cross faction/affiliation the cards? I know you need to keep the series together, but can you have an Imperial Affiliation series (1-6) with a Jedi series (1-6) in the same build?

You cannot have Dark side and Light side cards in the same deck. That's why they have different backs. However, you can mix affiliations from the same side, for instance Imperial Navy with Sith or Jedi with Smugglers and Spies. Some objectives specify "X affiliation only", which means you can have them in your deck only if you have the named affilation.

Khudzlin is correct..

Keep in mind that with mixed affiliations, you have to watch your deployments a little more closely to ensure you have resource cards from that affiliation...

so if you have a SITH deck with several navy sets added... you will need to have 1 Navy resource/objective card in play to play any of the navy cards out into the table. Could be frustrating having a whole hand of navy ships with 3 sith objectives in play so you can't them into the action.

That's why you usually have the affiliation card that matches the fewest objectives in your deck: if you have 7 Sith objectives and 3 Navy, you can guarantee you will start with at least 1 Sith objective (since at least one the 4 objectives you choose from will be Sith) and your affiliation card will give you a resource match for Navy cards. If you like it more risky, you can split 6-4, and no longer be assured of a resource match for you majority affiliation (though it's very unlikely you'll see all 4 minority affiliation objectives).

... (though it's very unlikely you'll see all 4 minority affiliation objectives).

Yeah - unless you're me in a tournament. Then it'll happen to you 3 times out of 4 games... Stupid Imperial Command...

That's why you usually have the affiliation card that matches the fewest objectives in your deck: if you have 7 Sith objectives and 3 Navy, you can guarantee you will start with at least 1 Sith objective (since at least one the 4 objectives you choose from will be Sith) and your affiliation card will give you a resource match for Navy cards. If you like it more risky, you can split 6-4, and no longer be assured of a resource match for you majority affiliation (though it's very unlikely you'll see all 4 minority affiliation objectives).

This isn't always the case anymore depending on your meta.

If Snoova and the hunters are a big part of your DS meta it may be wiser to find a 6/4 split where you take the affiliation of the 6 objective faction, knowing the 4 affiliation cards may end up going to edge if (A) your opponent is chaining Snoova early or (B) you end up resource locked. It's not common but if you really can't afford to have your major units Snoova'd and you still want to mix affiliations I think it is more viable than ever.