Moish, Everyone gets judged based upon where they are. The point of the moshol wasn’t to make you feel that you are so far removed (which you aren’t) but rather to show you that the main focus in life is spiritual growth. A yid needs to be growing spiritually. This is difficult both when things are hard, and when they are easy, as the dvar torah explains.
As you have stated, there was a “regression” spiritually, but that’s in the past. When a boy climbs up a ladder, and all he does is look down (or remember his previous falls) he will not get very far. However, if he look up at all the climbing he can still do, there will be a greater distance traveled (If my memory serves me correctly, there is a story about R’ Akiva Eiger to that effect)