In short, neoclassical economics is the reigning orthodoxy. It centers around modeling as a method of understanding the world. Its major divisions are micro and macro. Macroeconomics does include both Monetarist and Keynesian models. It should be pointed out, however, that there are other Keynesian views which are considered unorthodox.
Optimization is a modeling technique. For example, a utility function is stipulated for an economic agent and solved for the maximum utility bundle of goods. A rational economic agent is assumed to be one who would choose the utility maximizing bundle of goods instead of a sub-optimal bundle.
Why then Austrians reject such optimization, would Mises not say that maximizing utility is the basis of human action? Is it because such models assume some too large degree of objectivity, that in effect utility is less subjective then it really is? Is that the problem?
It’s more than just the rejection of mathematical formalism, but that’s part of what distinguishes Austrian economics from neoclassical. As Mises pointed out, mathematical functions cannot be used in economics theorizing because there are no constants in quantitative relationships generated by human action.
Take a look at Human Action, pp. 347-354; 706-711.
More generally, Austrians accept human beings as they are without the reductionist assumptions typical of neoclassical economics. For example, Austrians accept genuine uncertainty as an integral part of economic theory.
While Austrians reject mathematical formalism (and hence, optimization), they don’t reject the logic of economizing. Because of this, Austrians and neoclassicals can produce similar conclusions. For example, both would argue that production in the market economy is regulated by profit.
Thomas Taylor’s monograph provides some highlights of the distinctiveness of Austrian economics: