OP - you may think you nearly got arrested but if you had been it probably would have been a wrongful arrest, because:-
1. The Air Navigation Order (ANO), which is enforced by the Civil Aviation Authority (CAA), places limits on where a drone can fly. Breach of these limits is a criminal offence enforceable ultimately by a public prosecution.
Your description of your flight suggests that you were complying with these limits and therefore you were not committing a criminal offence.
2. Individual property owners can place restrictions on the use of their property for take-off and landing. For example, Dover Harbour Board can prohibit the use of their land for take-off and landing of a drone. Breach of such a prohibition will be a
civil matter, probably an action for trespass, unless the organisation imposing the prohibition is using statutory powers. Usually if the police get involved because a landowner has complained you can point out that you have not committed any criminal offence provided the ANO limits have not been breached. I don't know whether you were taking off from or landing on Dover Harbour Board property, or whether DHB have adequate statutory powers to impose a prohibition, but
if you were not using their property and you were complying with the ANO there is nothing DHB can do to stop you.
3. The reason DHB can't do anything is because owners of properties which are not excluded by the ANO cannot prevent you over-flying their properties provided you are at a reasonable height and you do not make such frequent over-flights that you could be guilty of harassment. Here are some excerpts from a legal briefing note prepared by the Country Land and Business Association Limited: 'However, for non-commercial drone pilots that do not respond to a polite request the answer for now seems to be that aside from a potentially costly action for an infringement of your privacy or trying to convince the police that the flights amount to a criminal harassment
there is little you can do to prevent them....
Section s76(1) Civil Aviation Act 1982 prevents claims being brought in trespass or nuisance against aircraft, provided they are flying at a reasonable height and comply with the Air Navigation Order....
Landowners in England and Wales are not entitled to all the airspace above their land, the position is well summed up by Griffiths J in
Bernstein of Leigh v Skyviews & General Limited [1978] 1QB 479.'
I have printed several copies of the CLA briefing note and I carry them with me ready to hand out f anyone accosts me. You can download it here:
https://www.cla.org.uk/sites/default/files/GN10-17 Drones and Private Property 2017.pdf