I've personally not ever had a genuinely 'angry' person question my flights. I have had curious people approach me, and only one or two actually question the 'legality' of me flying at a given location. More often than not, they want to know more about the drone, and what it does or can do, than what I'm actually doing with the drone.
However, I did discover, quite accidentally, that if people 'believe' I'm flying with a 'purpose'...which to them, seems to be something other than 'flying for fun'...they typically lose interest. I've never lied to anyone about what I'm doing, but as in any other situation, I've realized that by carefully wording my responses to their questions or comments, they frequently will infer that I'm doing something...'important'...something other than 'having fun.' Or spying on people.
An example: I fly occasionally in a small, local, nature preserve. There are numerous biking, walking, and jogging trails winding through it, but zero signed or otherwise clearly visible prohibitions on flying UAS. (There's nothing on the local municipality's website prohibiting UAS operations at that preserve...nor any others under their jurisdiction...I checked before my first flight.) Recently, a couple biking in the preserve area stopped and wanted to know why I was flying there. They asked, "isn't it illegal to fly in a park area?" I told them, "It is in national parks in the US. But, I've flown multiple flights here in the winter and am flying now to shoot video and photographs so as to compare the amount of tree leafing and water levels in the swampy areas as to what was there during winter." (Which is one thing I will be doing with the video; some before/after documentation...but only for my interests...and in reality, more for editing practice than anything else!) I was greeted with a blank stare for a moment, then smiles. "Well," the woman said, "that's cool. We love coming here and anything to help keep it natural is awesome! Have a great day!"
And off they went...