If i play Neutral Faction as my house can i you cards that have House X only deckbuilding restriction. I guess that not.

No you can't since your not that house, your house alignment is neutral.

inb4 'Unless your running the Shadows agenda and using "House X Only" cards that have the shadows symbol.'