Private fields are like your private home. You DO have the right to refuse anyone's attendance. It's your Legal Right.
Fair? It has nothing to do with fairness or discrimination. It's YOUR land. Now if you want your rights to be put aside, so be it.
I dont understand what you find so hard to get in all this.