The only things I can think of to ensure RTH works properly is to make sure you set the RTH height higher than any possible obstacles in the area and make sure you have a good GPS lock so the home point gets set correctly.
To test, just take off, fly out a comfortable distance, then hit the RTH button in the app and see if it works.