This isn’t really like what you said, “We all use the same roads” sharing it with bikes and pedestrians. The closest comparison using roads is a race track, or the autobahn. If you were screaming down the road at 180mph and some idiot on a skateboard pops up in front of your new Ferrari and it’s too late to correct your course, who’s at fault? Wouldn’t you hope someone would have some rules in place to minimize YOUR chance encounter with someone who might not think everything through?
I don’t really LIKE the fact I have to drive 3-6 miles from my home in order to fly, but I understand why there are rules. While, I trust, most remote pilots have responsible intentions and don’t go out trying to cause trouble, there are others who probably do what they want, no matter the cost to them or others.