R’ Avigdor Miller Z’l is reputed to have said “if yeshiva bachurim would wear yellow hats, so would I”.
Clothing identifies you as part of a certain group.
With some of the replies I am reading here, I am seriously reconsidering my black suit and white shirt (and yes, black hat) on shabbos. If the team that wears this uniform can express such a snobbish, horrible, attitude, I think I want to switch uniforms and be identified with another team.