Why should that scare you if you have high self esteem?
If someone’s self-esteem is high, he doesn’t feel a need to compare himself to others, and even if he does compare himself to others, it doesn’t bother him if they are better.
Besides, you are obviously better than them in at least one area (the area of self-esteen).