The both cancel claim and have a new effect, though without the word "instead" are they not technically claim replacement? They cancelled claim, and then have a new effect?

The curiosity comes from, since they both cancel claim, can you play 1 vs the other since, technically, it seems claim is cancelled after the first one is played - leaving nothing for the other to respond to.

Such as: Player 1 wins a MIL challenge against a Martell player (P2). P2, is the first player and gets first response. They play Red Vengeance to cancel the claim effect and initiate a new effect.

Can P1 now play BSobG?

The first player gets to play their cancel first. If it is successful, there is nothing left to cancel and the other player has to sit there with their thumb up their ass.

My brain is fried today so I didn't really read the scenario, but either way, once claim has been canceled successfully, it cannot be canceled again. Obviously claim replacement is different because claim still exists after the replacement effect happens.

That is what I was thinking since it isn't real claim replacement like some other effects but I saw it being discussed on cardgamedb and wanted to double check before I chimed in.

How would this work with Pyat Pre ?

Pyat Pree is a replacement effect. There is always a claim effect to be considered with Pyat, so the discussion above has absolutely no application to Pyat at all.

The Red Vengeance and By Sword or By Guild question is better answered this way (without the "it's not a claim effect" reasoning): After the claim effect is canceled, the rest of the effect (opponent satisfies claim or 3-power stuff, respectively) initiates and resolves as part of the "then" effect for the canceling event. "Then" effects give you no opportunity to cancel them separately. So no one is going to be able to use a second "cancel claim" effect before the "then" part of the first one resolves anyway.