If anyone could clarify this for me, it'd be appreciated.

I can't recall specifically which investigator I was playing, but I drew an other-world encounter in which "a monster appears!" From the cup appeared a dark young. At the time it spawned I had approximately 4 sanity, and 4 stamina. I sized up the fight, and I don't think evading was an option (it had a -2 to the check, and my sneak happened to be 2). Based on this I chose to fight. I passed my horror check, and had no issue with the nightmarish. It was about this time that I noticed the physical resistance, and despite a heck of a lot of bonuses, I still managed to botch the roll and not get the 3 successes needed.

I understand that because I didn't get the 3 toughness in one shot, I 'lost' that round of combat and take 3 stamina loss. However, because the monster spawned as a result of an encounter card and no gate came with it, at which point does it return to the cup?

Or put another way: my investigator was not knocked unconscious, nor was the monster defeated after it appeared. Does the monster return to the cup in this instance after I take my 3 stamina, or does it remain until I either a) evade it, b) defeat it, or c) it defeats me.

"A monster appears!" encounters are handled exactly like monster encounters during movement . That is, if you're still conscious after failing the combat check, you have the options to fight or flee, as normal. The monster does not return to the cup unless you evade it or it defeats you, or if you defeat it but you're not allowed to take it as a trophy for whatever reason (Endless).

So in your case, you'd have to do another combat round, or opt to flee (but you need to spend clue tokens to even get dice for the evade check).