Ok, Say you have two people. person A wants money and is beating person B who is trying to charm, negotiate and coherce person A to not take what they have on them now.

am i just switching between the two encounter type depending on the turn? I guess if person A was a group i would have to use leadership. So any ideas?

I think in this case, I would just have the acquiescing character make a one-time check. That is, you're in structured time. Let's say the attacker goes first and makes a Brawl check, succeeds in doing damage. The target decides to surrender and give up all his money. You could either just have combat end there (with no social check necessary), but if you wanted to emphasize the fact that the attacker is in a rage or something, have the surrenderer make, say, a Charm check to beg the attacker to lay off. Just make it a one-time roll, perhaps with boosts if the surrenderer mentions giving up whatever it is that the attacker wants.

Where the lines blur most is with talents like Scathing Tirade. This talent trio lets your social character do strain damage to foes using Coercion. But I don't think I'd run things that way in the example you gave of the mugging or shakedown. That's a more simple case of "OK! I give up!" versus a slow whittling away at the attacker's resolve.

I make it a coercion check with some boost dice because you are causing physical harm.

I've actually run into this situation. Social encounters can operate within structured encounter rules (the book even discusses this) where the attacker would use his actions to well... attack. While the social character is using their action to make social checks. Either the attacker reduces the target's strain/wound low enough to incapacitate/kill the social character. Or the Social character is able to bring the attacker to a compromise (half strain) or make the attacker capitulate (strain threshold exceeded).

Boost and setback of course being applied liberally based on circumstance.

In my pirate fantasy game the crew was jumped by Wraiths in a thousand year old coliseum. The captain decided to try using social skills to calm down the wraiths, while the wraiths were trying to kill them. While the rest of the crew just defaulted to regular combat; the captain was able to give the wraiths pause (generate advantage and deal strain) to help the encounter. It was also a ton of fun describing how the Captain was dodging attacks from the wraith while having a conversation. Or getting hit, but using the threat generated by the wraiths to get in close and try to learn why they're here.

There are mechanics you can use for this in page 123 of the CRB.