Not true, because what others think of you will greatly affect how you are able to live your life.
If you are well known to be a thief or a murderer or other such things, it is likely that nobody will be willing to trade with you, with the possible exception of other thieves and murderers. Doesn’t sound like a very pleasant life to me.
Also, I’m no expert on objectivism, but I have read Atlas Shrugged and the characters did not seem to hesitate to use violence in self-defense. If you attempted to assault or steal from an objectivist, they would likely defend themselves and their property against you.