Sorted it out, after a whole day of trial and error. I reloaded the firmware, reloaded the default parameters from the website, did an advanced calibration for the phantom, advanced calibration and then software calibration for the transmitter, now it behaves normally.
It seems if you somehow...