In close quarters I wouldn't turn off the controller though, especially if I'm in a situation where I don't have GPS as backup - I had this exact scenario yesterday, taking pictures of our holiday chalet in a very narrow gorge, about 40 metres wide. The trees & roofs were having a strange effect on the breeze, causing a slight chop. I took off without GPS (this was just messing about in a fairly confined space), and there was a little bit of buffet, but nothing too serious.
Then the breeze picked up & my phantom started to drift towards a massive tree. The instincts from flying the nano around the living room kicked in; I applied yaw until the phantom was facing me & gently applied forward pitch. No drama. This is the easiest method if the phantom is near enough for you to see the orientation, because if it's pointing towards you, forward = return. Plus, I couldn't really outwardly panic as I was demonstrating the phantom to the site owner!
If you do find yourself in a panic (and you still have wriggle room) it's best to 'test' an input with small movements, gauge the reaction, and if it works, keep using it. Intercept the situation early so you have time to deal with it.
I've seen a lot of threads on here where people recommend buying a cheaper quad to practice with - this is the best advice of all.