I've had drifting as well, but it has always been with wind. When I fly it indoors, then I don't have the drifting issue.
If it's the weight, it could still influence the drifting as when it tries on compensate for wind, etc, it could not be using enough power given that it weighs more.
I did...