We find this Hanhaga mentioned throughout the generations that Jewish attire should be different. However, let us not forget that it did of course resemble the general type of clothing. Just as today, we all year shirt and pants, but the pants are dark and the shirt is white.
I must mention that I did find instances where Gedolim where not recognized as Yidden. That would seem that the clothing was not different. But perhaps with the above explanation it is not a Stira. Yiddish Levush is separate but not necessarily does it stand out.
Please realize that nobody Paskened that we must wear this color or that kind of hat. The idea is that we dress like a Yid, whatever the Yiddish dress currently is. Over time things do change. First those who don’t care about Levush take on the current style. Then, businessmen dress according to business standards, and so on, until the greatest hermit will finally button his shirt.