The kids who are called nerds are not called that because of their clothing. Their clothing is just an object that is used against them. Just like kids make fun of people’s name. You can have one kid who has a “funny” name and no one ever makes a joke about it and another with a similarly “funny” name who gets ridiculed all the time. Deal with the issues, not the manifestations.
The OP is 100% right. And if the community “is suspicious” of you, then you should definitely take the high road. Out of there.