def set this to NAZA a while back because I have used HL function so many times when I have had the copter out of sight and brought her back using HLAssuming the 'copter was facing away from you at takeoff (which is common, I would expect), in both HL and CL, the 'copter should have come towards you when you moved the right stick down. Moving a different direction implies that either you were not in HD or CL mode, the home point was wrong, or there weren't enough satellites.
Since RTH worked, it seems like you weren't really in HL or CL mode. Check the switches in the RC or Vision Assistant and make sure NAZA mode is enabled.
-- Roger