As title states, would it be too imbalanced to play the first quest with only three characters?

Yes, the first quest definitely is. With experienced players, three heroes is where it really is balanced. The Second quest is possible with 3 heroes as well, though they will get crushed if they don't know what they are doing. All the other quests are nearly unplayable with 3 heroes.