In New Hampshire along the shores of Lake Winnipesaukee, the problem you described happens regularly. Ultra-lites and seaplanes fly over the tree tops in such a way that you hear them before you see them.
This prevents me from flying within 500' of the tall pine trees. Because, while I can fly my drone, as you did, up to 200' to clear tree tops, I can't see over, and beyond, the tree tops. So I need to assure a VLOS of the air space I'm flying in for at least 500' feet. Then, I find that seaplanes that are ready to land, or ultra-lites, don't like to fly high. I can'd descend rapidly. So, if I see a plane at the last minute, I have 500' to rapidly increase my altitude to just below 400'. Better yet, I don't fly on really nice, low wind days on weekends in order to avoid the risk of manned aircraft flying at the tree top level. It seems unfair ... but prudent risk avoidance is my only choice.