Walter Block gets into some of this in the Q/A section of this playlist. http://www.youtube.com/playlist?list=PL879B7795698FED32
There is an interesting aspect to questions like this. If we could say definitively how something would function in an entirely free market and we could say definitively that is would function better that way than the current system, then we could centrally plan it that way. But the entire argument against central planning is that central planners do not know what would work best. If a monopoly of coordinated police/court systems is what works best, then that is what would likely arise voluntarily in the market. If it is competing private courts and security firms that would function best, then that is what would likely arise in the market.
We can philosophize about how it might function or other things that could be done (and that is a good thing to do), but we will never know all the ins and outs of how a completely private system would function until we allow the market to work out the details. But we can probably say this: If it requires force in order to fund a certain method of providing police and courts, then it is very unlikely that this is the system that is demanded by the market.