I always thought it was because people feel like they were being measured in terms of “you’re X old, and that’s all you have to show for it?”
My take? The smallest among us have already accomplished greater milestones (learning, commitment to yiddiskeit, chesed, ect) than the the average nocri can expect to achieve in a lifetime.
We have nothing to be ashamed of, considering our scorecard.