Highest winds I ever flew in were probably around 25-45 MPH sustained gusts. It was right around this time of year when you get crazy weather.
I usually always fly in GPS mode so I remember seeing my P3P being pushed off it's stationary hover point by a big wind gust --- then watching it slowly return to the original location once the gust subsided. But I do remember being scared because you never really know how big a gust will come along and how long it will last --- so it's possible a long, sustained wind gust could push your Phantom hundreds of yards away and you get this icky feeling like you might never see it again !! You just hope and pray you get a lull in the winds and RTH works as it should.
Basically, I've become very conservative with flying my P3P and never fly it when gusts are above 25 MPH.