A few things to consider. If the RTH is triggered by a total signal loss after 3 seconds, and not by the pilot pressing the RTH button on the transmitter, no stick input can be registered by the aircraft to alter the prerecorded RTH settings recorded on the aircraft before launch, until signal is restored.
Therefore, if the stick input was received by the aircraft, you had control again, and should have been using full throttle up to recover from the RTH, and immediately cancelled the RTH by pressing once briefly on the transmitter RTH button. If you then, again, lost full signal for 3 seconds, RTH would engage again, and you can repeat the process, as necessary to regain full manual control. If the app is ever asking you if you wish to cancel RTH, you have full manual control again, and should cancel, to avoid the very problem you encountered.
The goal of RTH is to regain signal control. It worked, but you didn't cancel it and take over. You are always smarter than RTH, once you have control again.