Remember that anywhere in the USA only the FAA controls the airspace. Other entities such as national parks, state parks, local parks and private property owners can control only what you do ON their property, not what you do OVER their property. Just don't take off, land or control your bird from the prohibited property and you're good, legally. That doesn't stop some AH from giving you grief and spoiling your enjoyment.
I've been hassled several times when I was flying legally from a public place. One time was so bad that a crazy guy was physically crowding me and waving his arms while shouting at me. Now I carry pepper spray for self defense, and I make sure to have my cell phone in my front pocket where I can turn on video recording in case of an altercation like that. The video could be very important in case I actually use the pepper spray and need to prove it was in self defense, because if not it could be a crime.
I often take off and land from someplace without other people around just to avoid hassles.