Popa-
This question has gotten me into quite a few arguments lately. Funny thing is that from my limited knowledge the only gemara I can come up with that is explicit about a specific dress code is the thing about not wearing red – which you said was only a chumra! It’s in Brachos 20a according to Rashi’s pshat in the story of the lady with the ??????, though interestingly enough I found the Otzar Hage’onim brings down that it was a shaatnez sheila (!) which actually seems to fit better. Anyway I hope to do the proper research and write a shtikel on this sometime soon.