I'm struggling to see how to fit Yoritomo in. His stats when non-boosted are terrible for his cost, so you definitely don't want to pay for him outright. Obvious choice is to Charge! him in, but then you'll need to include various other 4 or 5 costers to increase effectiveness of Charge! But if you don't have Charge! in hand at the right time (or if it gets discarded at the wrong time) then you'll end up having to buy your decent characters outright leaving very little surplus fate. Bear in mind too that Charge! and Gaijin Customs cost 1 fate each - I think the idea of Yoritomo on paper is better than it'll be in reality, although we'll have to see.
I think Kudaka will be a much better neutral option for us to use