Does the comet cancel out a chaos symbol and vice versa? I couldn't find it in the rules but it seemed logical to me.

Per RAW, they do not. Boons and Banes do; Comets and Stars can both come up and be applied on a single roll.

I have houseruled that they cancel, but I also count Stars as both a Boon and a Bane if one is rolled with no applicable effect on an action card, in order to make rolls less likely to succeed.

Uh if they are a boon and a bane don't they just cancel out?

Sorry, I meant a Bane and a Challenge.