My strongest experience of drone hate came from my own hood too. This was after years of flying around with no issues.
One day a guy went ballistic. I thought he was going to take a swing at me while I was flying. I got down ok and just went inside. Next he was posting on the hood Facebook page calling me a pervert.
The last thing I would think about any of my neighbors is that they were a pervert with out proof. I tried to show him the only thing I shot was the sun, but there was no reasoning with him. I think people project there thoughts on others. My armchair psychologist theory is, folks like this guy have perv thoughts so they think everyone does.