You need to ARM the motors by pulling BOTH sticks to the bottom of any corners first.
DJI called it CSC for Combination Stick Commands.
It is on page 13 of the manual below:
I have found that if I land the Phantom and let the motors shut off, or if I activate the RTH mode and then try to restart it again using CSC, it doesn't do so right away; seems to take a minute or so for the electronics to catch up.