You played correctly. Encountering the Black Man is essentially trying your Luck. You can't really defeat him, but if you're Lucky enough, he'll offer you some help. If you're not.. well, you were a good supper. Generally speaking, you start any combat by testing Horror. The wording on the token suggests that before anything starts (normally: Combat), you have to test Luck, and if you pass, no combat takes place (and neither if you fail...)
You got a strong party. Did you play with PS as well?
Man, I'm feeling such a deep nostalgia for the time I had to start discovering the game... It's so good to have new forces joining the defense of Arkham-Alamo and reading your stories