In Rogue Trader, during character creation, some characters can get the Rival and Enemy talents. In The Radical's Handbook , the book mentions these talents can be negated by spending XP as an Elite Advance (or by roleplaying and approval from the GM). However, is this the same in Rogue Trader, or are they stuck with those talents?

By RAW, there are no rules in Rogue Trader to remove them. Whether your GM/you decide to use the rules from the Radical's Handbook is entirely up to you/him.

I'd generally say that you should have to roleplay repairing your reputation with that faction before buying off the rival/enemy (or downgrading it).

If your GM is doing his job, those rivals and enemies should be a source of fun for you and your group. You will always have enemies and rivals... if you buy them off, the GM is just going to give you new ones

My recommendation is to work with your GM on developing those rival NPCs and enemy organizations. Have some fun with it.

Theres a very simple way to get rid of rivals/enemies, it's called: Orbital Bombardment

Personally as a GM, I don't allow my players to buy off enemies with xp, they can however buy them off with money, threats, etc etc (Roleplaying).

Like Badeblade said, however, Rogue Traders make new enemies by the wagonload, so there's always a fresh one in the waiting.

I think a combo of both is best, but sometimes "bought off" for free.

Like in one game, my PC came from an Ork infested world and she was in the last large pocket of human resistance. She got the talent, Hatred: Orks for free. And because of that, I make sure I RP it out fully. Including the joy of letting Hecutor Plasma Broadsides rain down upon Orks on the planet below.