The big difference being that the accelerometer and gyro (collectively the IMU) aren't vulnerable to external variables (except for temperature and voltage). Gravity, F=ma, and the Coriolis effect are pretty **** static!
The compass, on the other hand, operates in the very messy landscape of magnetism which has some primary global tendencies but is otherwise subject to all sorts of localized influence.