When mainstream economists use the phrase rational self interest they usually mean that a person uses suitable means (that’s the rational part) to attain ends that he perceives as beneficial (that’s the self interest part). Adam Smith famously postulated that we owe our supper not to the benevolence of the butcher or baker but to their self interest. In other words, in the market people act to benefit others out of regard for their own benefit.
Mises pointed out that this distinction between self interest and other interest isn’t relevant for the economic theory of human action. All we need is the distinction between the higher value of the chosen alternative and the lower value of the alternative not chosen.
Take a look at the section in Human Action on Rationality and Irrationality, pp. 18-21.