Another idea I'm toying with is a Noghri who turns his back on being an imperial assassin. If be playing the archetype often found in martial arts movies of the deadly but reluctant warrior. The almost pacifist, wide trying to right guys wrongs of his past life. Part deadly combatant, part Zen monk.

How you play the character doesn't really tie into what the character really does. You can build the PC any way you want, and you can play it this way. Bounty Hunter/Assassin or Spy/Infiltrator are both valid builds, but it depends entirely on what you mean by "assassin." Soldier is a good option for a sniper/special forces commando type. For raw strength, Marauder offers the best build for pure damage with melee/brawl. Bounty Hunter will be getting the Martial Artist spec soon.

I know, that's why I put this question out, there are many ways to do it. The martial artist is gonna be a focus if it turns out good. But what about the hermit-y skills?

"Hermit" literally means "any person living in seclusion". So that portion can be anything from Doctor to Mechanic to Survivalist, and beyond. Maybe the character became a Doctor as means of atonement, or learned how to repair landspeeders as a job, or is simply living in the wild and has trained their Survival skill up. Etcetera.

Ok your right hermit could be interpreted as recluse, but I'm b talking classic hermit, Zen monk in a cave. Survivalist sound good any others?

I'm gonna go ahead and second Doctor, if only for the fact that they can eventually, through the Pressure Points talent, make Brawl attacks that ignore soak, deal strain damage, and add ranks of Medicine, which would be great for a pacifistic martial artist type of character. You could probably even roleplay his actual use of Medicine as relying on hitting pressure points, acupuncture, or other flavourful twists within reason, or try to fluff your medical kit as collected herbs and the like.

Well, there's always the Hermit tree. Soldier Trailblazer might also fit the bill, though they get some party support talents that might not mesh with the concept. Really, though, anyone who has Survival as a career skill and a relatively calm demeanor should work. Since this is a party based game, there aren't really any specializations devoted to isolating yourself from everything and everyone.