During game setup the number of resource tokens is scaled with the number of players (1 resource per player) in the game.
For the buidlings the rule doesn't state a similar scaling (or I didn't find it). In consequence during the 2-player games we have played
so far, each city had 7-8 buidlings (up to 4 harbours) which significantly accelerated the game. For example it was possible to play a research card each round (even tech 4 cards could be played without gathering trade over 1-2-3 rounds). Due to the acceleration the game has less rounds and less movement phases for the armies and a military victory seems to come a little bit to late with regard to a tech or coin victory (all 2- player games have been "far away" from a military victory even though a player tried to conquer cities/ the capitol).

Does anybody made the same experiences? Does anybody already made some house-rules with less buidlings when playing with less people?
Or is it the same in a 3-4 player game?

Yep. this is mostly since you have free reign over the buildings if there is only two of you. You're not supposed to scale them down, but if you wanted to, I could see taking away one of each building for each player missing from a full game. So, a two player would take two of each building out. Just an idea, but there is no rule for it. I like the effect it has because it makes a different type of game with different players. Instead of just the same game with two less people sitting at the table.

Another alternative that echoes the computer game is to limit each city to have only one of each type of building (i.e. one harbor, one market, etc.) this will most likely add some time to your game, but opens up the number of components/buildings for everyone to use...

We've played it this way and still been able to produce large amounts of trade and Modern wonders...

I have to disagree about limiting the number of buildings being built. The Russians start w/ 0 building techs. The first one I went for was navigation so I can build on my water tiles. I had to keep building that until I had enough to build without cashing in trade to build. I of course went for speed to steal tech next. But I found no issue on traveling across the map for a military victory was reached before any other.

My scenario was Russians vs Romans. I was the russians. The first thing I could build was Stonehenge since I had to use my scout to gather extra building resources. I later built 4 ship yards from 2 cities because of my lack of technology, but ample building resources. I ended w/ a military victory because my opponent went culture misplacing their 2nd citing in high trade, but low building area. They should have transitioned to tech victory, but had more culture tech and less military allowing me to march into their capital undefended.