Let's say you have a situation like this where two separate conditions might call for both a Setback and a Boost die...

The PC wants to shoot a guy. It is outdoors, and it is night time.

GM adds a Setback die because it is dark. GM adds a Boost die because the target is standing under a street lamp.

Do you add both dice, or does the one kind of nullify the other one and you add neither die?

You add both die and see how things work out. That's the fun of the system. Also note, they don't have exactly opposite results, so they wouldn't cancel anyway. Although as below, this might not be a situation to add any dice.

Same as how you can three despair results and still pass a check. Or three triumphs and still fail a test.

In the situation that you have stated I wouldn't add any boost or setback, since even though it's night, the target is clearly illuminated. However in general you would roll both any additional boost and setback, otherwise aiming would remove setback dice, rather then add a boost. I think there's a paragraph on how boost and setback dice don't cancel each other in the core book, but I don't have it readily available now.

No, because a Boost die isn't equal to a Setback in regards to Advantage/Threat.

Boosts and Setback do not cancel. This is because of the fact that they won't (necessarily) roll the exact same result. You might get a blank on the boost die and failure on the Setback, or Success on Boost and nada on Setback, or anything in between. Adding a boost and a setback broaden the chances for higher success or greater failure, but they don't intrinsically cancel.

There are several Talents or pieces of gear that nullify Setbacks, however.

I agree that the OP example wouldn't give you any setbacks or boost dice. If a situation that incurs Setback is negated , then the Setback goes away.

However, in situation where you get both Setbacks and Boost dice (say, you're in dim lighting which incurs a Setback, but your target is wearing brightly-colored clothes or being really loud), then you would roll them both. So, Boost and Setbacks don't cancel each other out, but situations can be negated (take away the Setbacks) and/or mitigated (add Boost dice).