The purpose of the barometer is mainly to hold alltitude by sensing change in the barometric pressure, temperature and air pressure. This and other sensors sending info to the main controller is what keeps the quad holding at a specific alltitude for hovering. In 90% of the time it will not give an accurate ground level alltitude. If your craft is holding alltitude when you hover then the baramoter is working and don't worry about the readout.
When landing whether it be on the top of a building, on the ground, on a boat or the side of a mountain the quad sensor determine that downward movement has stopped and not that it has reached 0 alltitude. It's like us stepping down steps and getting to ground level. We don't know what the alltitude is just that we stopped going down and we are at ground level. In the mountains we may be at 2000 ft above sea level but on each mountain or hill we are at a different alltitude. 0 is relative as to when we are touching ground. Hope this makes since.
When the craft is up in the sky its controller uses location information and telemetry data from GPS to get location and alltitude information based on triangulation and orentation. Just fly and have fun, the onboard sensors and master controller maintain all the data need to make all those calculations for us.