Just went out and tested this issue, I seem to have the same problem. Assuming the altimeter in the Phantom 3 is based on pressure (aka a barometric altimeter), then these are the reasons I can think up as to why this happens:
- These types of altimeters are sensitive to temperature. As the temperature rises (as the aircraft warms up) or as the temperature falls (as you get higher and the temperature around the aircraft falls), the altimeter will give different readings. So wind, altitude, or anything that affects temperature might also affect the reading
- Sudden changes in air pressure (owing to wind, cold fronts, etc.) will cause your altimeter readings to vary wildly
Today I noticed that my Phantom will actually sink if left a few metres above ground. I tested this by aligning the Phantom with a tree in the distance in my line of sight, and noted that sometimes when it dropped, it wouldn't climb back up to where it was before. Within a minute or two, it was just half a metre from the ground after sinking about 2 metres.
However, in an earlier flight, I noticed that it remained pretty much accurate throughout the flight (since I happened to notice negative altitude readings as I began to fly it down a hill I was on, so it must have been more or less accurate). This particular flight was the second of two consecutive flights, which leads me to believe that it is based more on the aircraft warming up than anything else.
An altimeter of this kind is not really supposed to give absolute readings - instead, its primary function is to detect relative changes, so the Phantom can tell if its going up or down. As such, you can expect the reading to be off after flying around for a while, as the error begins to build up and the environment changes.
As for the VPS, it should always be activated, but it only starts to limit decent when it detects you are very close to the ground. It should override whatever the barometer says, and that's what I noticed today when my decent slowed right down about a foot from the ground, despite the app reporting I was 10m high.
Regardless, if it bugs you, there's a way to zero it again. If you fly around, and let the aircraft temperature stabilize for a minute or so, then bring it down and land, the reading will be set to 0 and you can take off again with (hopefully) minimized error in the altitude readings. The reality of it is though, these machines are for aerial shots, so if you're a couple of metres out from what the app reports and you're a hundred metres in the sky, it's usually no biggie