Morality can easily be based upon the autonomous agency of rational creatures . The action of an autonomous being, when it impedes the ability of another autonomous being to self-govern, is the defining characteristic of an immoral act. This account is insensitive to culture or society, it is only sensitive to the nature of the interacting agents. It allows you to rule upon the rightness or otherwise of an act, regardless of whether that act is legal or not.
Except that it falls down pretty hard when applied to reality. There are inevitably situations where acting in either direction (or not acting which still counts) will harm one party or another. In that situation which action is right? Well neither are purely right obviously. But which one is more right? That inevitably comes down to a subjective decision. Even if that subjective decision is based on objective data the act of ascribing weight to that data is subjective.
As an example. I have X people and presented to me is a button that if I press it will make 1 of them very unhappy but X - 1 of them somewhat happy. Different people will push the button at different values of X. We can measure the level of change in happiness objectively and we can determine X, that can help inform making the decision to press or not. But deciding how much the larger groups happiness is worth vs. that one persons unhappiness is an extremely subjective determination.