Having now been observed by a genuine FAA Inspector while flying, I can say you should probably be within about 1200 feet of the aircraft on a nice day. It is certainly allowed to look away momentarily or even lose a direct LOS momentarily as long as you can re-acquire the sight of the aircraft and its surroundings very quickly (or you have a visual observer in direct communication with you).
Just chiming-in here...YES, there is a set of criteria for what comprises VLOS. While not very well defined for hobbyists/recreation, it is well laid out in Part 107.
§ 107.31 Visual line of sight aircraft operation. [my bold letters shown]
(a) With vision that is unaided by any device other than corrective lenses, the remote pilot in command, the visual observer (if one is used), and the person manipulating the flight control of the small unmanned aircraft system must be able to see the unmanned aircraft throughout the entire flight in order to:
(1) Know the unmanned aircraft’s location;
(2) Determine the unmanned aircraft’s attitude, altitude, and direction of flight;
(3) Observe the airspace for other air traffic or hazards; and
(4) Determine that the unmanned aircraft does not endanger the life or property of another.
(b) Throughout the entire flight of the small unmanned aircraft, the ability described in subsection (a) of this section must be exercised by either:
(1) The remote pilot in command and the person manipulating the flight controls of the small unmanned aircraft system; or
(2) A visual observer.
Seems to make sense to me.