The card says "While this Player is not President no other players may not be chosen with the administration location".

So. If Player 1 is president, Player 2 gets VP, Player 2 becomes president (via administration or anything else), then they (theoritically) hate him and take the presidency away via administration and Player 3 becomes president.

My question is : does Player 2 remains VP after that? (in other words can they target Player 4 with administration while Player 3 is president?)

There is nothing on the card about discarding it any time at all.

Yes, they still remain vice-president. So if Admin was again used to move the Presidency, it would have to target him/her again.

This always seemed wrong to us; our house rule is to discard the VP card if that player becomes President.