At the risk of sticking my head into a hornet's nest (I have observed that this is a sensitive topic, and for the record I am not saying that ops beyond VLOS are inherently dangerous or reckless - though they do introduce different risks that need to be considered and managed).
Generally, for the US, the answer is that the aircraft must be operated within VLOS. But there are multiple sets of rules . . .
For commercial ops, it is stated in 333 expemptions and in part 107 - both of which let you use a Visual Observer if it is beyond VLOS of the pilot.
For hobby / non-commercial, there is vagueness and confusion because of where the restriction is buried.
Based on how I read it, there are 2 items that come in to play for anyone considering hobbyist operation beyond VLOS:
1) The actual VLOS limitation is spelled out in the FAA reauthorization bill (Public Law 112-95) under section 336 "Special Rule for Model Aircraft." 336(c) defines a Model Aircraft, and 336(c)(2) specifically states that it is "flown within visual line of sight of the person operating the aircraft." So, if you want to be operating a model aircraft, you need to be VLOS - otherwise, I'm not sure what the FAA would say that you're flying.
2) If you decide to fly beyond VLOS, even though many hobbyists are not pilots, the FAA can pursue enforcement action if they feel you are endangering the safety of the national airspace system. Refer to 336 (b) for the exact text. So, if the FAA feels that you are a danger to airspace (and I'm assuming that they'd interpret that to include danger to people or property on the ground), they can pursue enforcement action against you. Enforcement action may not be practical or probable, but it is possible even against hobbyists.
If you are having problems falling asleep

, or just want to see the text for yourself, here's a link to the law and the relevant sections that apply to the FAA:
https://www.faa.gov/uas/media/Sec_331_336_UAS.pdf
It's a subset of the relevant sections from the law applicable to the FAA and some of the most riveting government reading I've seen of late!
BTW, if I have my facts wrong, please help me understand where (the US Gov and FAA do not make it easy for mere mortals to understand their rules.)