Catelyn stark does not have a "to" in her text so I was wondering if her ability is a cost or an effect. Can it be negated if so is she discarded from or return to my hand?

As you mention correctly, Catelyn Stark's ability does not have a cost. As long as the play restriction ("during an INT or POW challenge") is met, you can trigger the effect. If the effect is cancelled, Catelyn stays in your hand. Note that it's probably not worth cancelling the effect most of the time, though, because you could just trigger Catelyn again after the cancel.

Catelyn leaving your hand and entering play is the effect (as you say, there is no cost). So if that is canceled, she never actually leaves your hand - and stays there.

It's a little confusing because, in a practical sense, you have to show her to your opponent before s/he can decide whether or not to cancel, but the technicalities of the rules can take a little detour from common sense sometimes. When you play/trigger an event card from your hand, part of it does leave your hand when you show it to your opponent during initiation. And since it stops being part of your hand during initiation, canceling an event card makes it go to the discard pile. But, because she is not an event card, showing Catelyn to your opponent when you trigger her effect doesn't actually make her leave your hand - meaning that if she is canceled, she is still part of your hand and is therefore not discarded.