I occasionally check to see if an IMU cal is needed and it never has been, even after two updates.
I would set the RTH/Failsafe altitude to whatever is sure to clear every obstacle between you and your mission. Remember, the bird is going to want to come straight home and so may encounter something you normally might not. I would then add 10 meters to whatever 'safe' height I needed, just as a margin of safety.
I would not make the altitude any higher than that margin of safety because it requires power to climb. In an emergency you may need every amp you have for horizontal travel.
So the specific answer to your question "Is 100m too much?" is No, not technically in that the bird can still make 100m, but you might not want it to have to climb that high.
Edit: Wow, you've never set your RTH/Failsafe altitude? I believe the factory has them set at 70m so you've at least had that much. You have several other settings you need to make so I suggest you watch a few YouTube videos about how and why to do them. The videos are great learning tools.