Just practice with it a few times and you'll get over your fear over it, then you can use it at your leisure. Just don't get too comfortable with it and lazy about making sure the return path is clear.
In DJI's Wookong and A2 flight controllers, the go-home function (used in the Phantom's failsafe) is considered a useful enough feature that you can assign it to its trigger to its own control channel.