As you hint at above, everything in this debate hinges on definition. Certainly it’s possible to define “liberal” in such a way as to exclude people like Smith, but I don’t really see the usefulness of doing so. Also, if you read works like Milton’s “Areopagitica,” you see arguments for things like freedom of the press that do not seem to appeal to community interests.
Also, your point about utilitarians is a good one.