Sam, a Polo shirt is not unique to any group of Ovdei Hashem. At most, it can be acceptible.
Tzaddikim did encourage the Levush for the specific purpose of hindering the person’s free wanders. When you are constantly aware that you stand out as one belonging to a group that doesn’t belong in such places, you’ll feel akward going there.