You can't fly around mountains.
You need a line of sight from the transmitter to your Phantom.
Whenever there are obstructions between you and where your Phantom is, you want your RTH height above the obstructions.
Without knowing the answers to these questions, I cannot answer your question.
1) How long will the P3 fly it's last heading and, at what speed after loosing contact with the controller? I'm assuming you are using a controller that needs to maintain visual contact with the P3.
2) What kind of terrain are we talking about? Some of us have seen terrain which rises thousands of feet within a very short distance.
Find a programmer who can hack the P3. Make it return to some point before it lost contact with the controller. Once it reaches that predetermined point, allow it to initiate RTH.
Here's one even better that that...
Have your P3 notify you that it has returned to that point, it has been reconnected but will not continue as previously instructed, and is awaiting new instructions. If you do not input new instructions, it will initiate RTH.
Contact is lost the bird stops, backs up, and awaits further instructions or (depending on what you have chosen in the programming choices), initiates RTH from that predetermined point it has returned to "before" it lost contact with the controller.
Not to contradict Meta4, but... yes you can fly behind a mountain. Been there, done that, and it really sucks.
So to answer the OP's question I will suggest finding a programmer who can hack the P3...