Drift is unlikely to be caused by the imu directly. Position stability is the job of the GPS system. It’s information gets integrated with the imu, altimiter, and user inputs to control the drone flight. This solution is being updated many times a second to fly or hover. The conditions are always changing - wind speed, wind direction, gps signals etc.
In your case, it sounds as if you may be experiencing some effects of the GPS system. The GPS system position information is also getting updated multiple times a second. It will never have exactly the same value twice, even if anchored to the ground. The GPS satellites are in constant motion, and their time clock signals have some jitter to them. The normal system is only specified to deliver accuracy within 16 feet 95% of the time. Generally it is quite a bit better. This effect can easily be demonstrated by performing a few RTH thru landing with the precision landing disabled. Most landings will be within 4 or 5 feet, but rarely better.
Exactly the same effect can be seen in a steady hover where the drone drifts randomly in a several foot radius. Drones used for serious mapping use more expensive, more accurate GPS Systems to avoid exactly this problem. Purchasing one of these solutions would solve this issue for you