I personally go with always rerolling the correct amount.
Adding or removing a die after rolling is too complicated. What if he doesn't roll all the same result? Which do you remove?
No matter what you choose to do about it, be consistent with it throughout the rest of the match. Changing how you do something in the middle of an ongoing game is definitely unfair