Of course you can privatize the police and the justice system. All you have to do is sell the police and courts to somebody and there you go, that would be by definition privatization. How would you know a good court from a bad court? The same way as everything else, word of mouth. If someone was to be a bother and not abide by court rulings? Simply don’t do business with that person or company. Give it time and people will adapt to the circumstances.
Remember that humans act and will attempt to better their situation. If someone or some thing tries to interfere with that they will change their plans.