Did another IMU calibration and compass calibration this morning, VPS is on - problem's still there. Also, there was a VPS error popped up on the screen this time. So I turned off VPS, it was a little bit better compared to my existed problem wih unstable hovering like before. I guess I have to turn off VPS everytime now until a new firmware update is released.
The camera gimbal is still extremely stable. I managed to fly around and took awesome pictures at 15km/h wind at 120 metres. All is well except firmware issue.