Ok. Tried the Phantom today and it behaved the same way.
While I'm in GPS mode it stays still, very steady by the way.
The problem shows only when I try to rotate, I'm not convinced that it's the wind and my Phantom doesn't come back to the starting position.
I tried to rotate 45 degrees...