I’m inclined to agree with you. To be more specific I would be somewhere between a right-libertarian and a paleo-conservative.
That is not the only difference. Most libertarians don’t tend to have any problems with open borders and if they do it is mostly because we have a welfare state. Other than that they don’t see any problem. Also the libertarian view on abortion as expressed by Murray Rothbard is abhorrent to me although I don’t personally fault Rothbard for trying to think logically about what libertarianism implies. I just can’t support. Then again Ron Paul has argued against this interpretation and stated his own libertarian justification of libertarianism.
Also I should add that Paleo’s tend to be against legalizing drugs but I suspect they would agree with libertarians that the War on Drugs is bad.