Hi Bobby
I apologise now as I've skimmed through this thread so may have missed a few points. My initial thoughts are, 1 as Xitor touches on: communication.
Go speak to the guy, apologise, and explain what your doing and what the video / images may or may not show. OK, whether you have anything to apologise for is subjective, but it gives you a path to get your neighbour onside and ease his concerns.
Chances are, he doesn't understand much about the drone and assumes your snooping on him, for whatever reason. The fact that there are loads of high level aerial pics of his property online will not reduce his concerns of being 'spied' on by his neighbour. Even though in reality you're just flying for fun.
2. I'd generally stay away from flying within houses / estates / densely populated areas. Certainly if you do go above fence (and head) height. A flyaway / failure can occur for a number of reasons, without warning. And even a small Phantom in a slideslip into a childs head can be life changing. You don't want to live with that, trust me.
3. If you are in some form of social / council property (UK). You will find there are many restrictions and limitations to the tenancy agreement. These will include things like noise, anti social behaviour / the use of powered / noisy equipment and a host of other things.
So while you feel you've done nothing wrong or illegal, you may have crossed a line in these terms. Again, not worth the hassle or potential penalties.
Often these terms and restrictions use loosely worded language like 'reasonable' and other subjective words. When reviewed by people that do not understand (or like) hobbies like drones, reasonable turns into unreasonable very quickly.
4. The Police would have been round by now if they were called. And, if you had done anything serious, the CAA would have been involved alongside the prosecution service. The best advice here is: know your stuff! If the average Bobby knocks on your door asking about your drone & flying, chances are they wouldn't have a clue what a CAP 658 was.
BUT! There are a load of other laws that you could be infringing: Noise. Anti Social behaviour. Breach or the peace. Privacy. Data. etc. etc. etc. Some civil, some criminal.
Speak to your neighbour and try and keep your drone away from people as far as practicable.

NP