Yoren Question

Yoren can use a Ranger Patrol card to build a guard for free. How does this effect the Largest Patrol 1VP Card? If you have 3 Ranger Patrols in front of you and the VP card and then use Yoren does that mean you have to vacate the victory point because Yoren's card says to Discard a ranger.