The ones looking down at someone just for wearing jeans, is in the category of being judgemental, which all humans are. Being judgmental is part of our instinct and is healthy and good when used properly. If the person judging will act in a way that is not within Torah guidelines, then they need to reevaluate their premise in judging. If the judgement leads them to a place where they won’t ask that person for fashion advice, then the jeans wearer should just chill out, and not feel like every person that doesn’t wear jeans is looking down on them.
I do maintain though that choosing to wear jeans, even if just for comfort, will put a person in a certain segment of society, where clothing choice does categorize who we want to be seen as. Same with a person who will ONLY wear a white shirt and black pants. They want to be seen a certain way.