I don't have any experience flying a quadcopter at that sort of altitude but have many hours flying full sized aircraft and the principles of the effect of altitude and density altitude (google it) remain the same for any aircraft. The flight surfaces ie the props and therefore the motors will have to work harder to generate the same lift as at sea level. Expect reduced flight times and a reduced response when manoeuvring, particularly on hot days. You are very sensible seeking anecdotal evidence of Phantoms flying at that sort of height as (to the best of my knowledge) there aren't any performance graphs to refer to. I suspect it will fly adequately as it is such a nimble and responsive aircraft at sea level but don't rely on that.
As for safety, stay below 1000 ft above the highest object in your local area (I'd stay below 400ft above it personally). Stay clear of airfields and places where you regularly see helicopters.