I dont get the point of occupying a castle, you really need to have lots of units in it to be able to hold it and will always have a great disadvantage when fighting against an army with a siege tower. AND the aspect of siege towers having no defense strange which is quite fine btw, leads to another issue, that it is easier to leave the castle and just retake it instead of trying to fight for it... This problems could be solved if castles apply a +2 or +3 defense the garrisoned units so a siege tower just give a slight advantage and after taking the castle you get a proper boni for holding it that could solve a lot issues...

The point of occupying castles is to win the game.... If you kick someone out and keep pushing you're leaving that castle open to be retaken by someone from another direction that could result in them getting their 7th for the win.

also,

castles and strongholds are the only place you can muster more units.

Some cards as well (Theon comes to mind) also get a bonus for being played on Castle territories.