Sam, that seems to be a misreading of the Pasuk. Those 120 mentioned in the end of Bereishis are referring to the time from the warning until the flood. That’s how the Targum and Rashi explain it. And you see that it didn’t go down to 120 until Moshe Rabbeinu (who interestingly is hinted to in that Pasuk).
Until the flood the upper limit was a thousand years, after the flood it was halved, and after the Haflaga it was halved again and continued to go down — at least for Avraham Avinu’s family.