I’ll go with bein_hasedorim’s reason about the men.
About women’s clothing, this is fairly new. I got married it the late 70s and everyone wore colorful clothes (not red, of course) from the most Chassidish to the most Yeshivish. It was concidered incorect to wear black. I think the black thing started when there were many affairs one after another. If you spilled something on a turquise dress, it showed and you didn’t always have time to go to the cleaners. On black, it doesn’t show. Also if everyone wears the same color, then people don’t notice if it’s the same dress 3 times in a row or not. Colorful dresses, you propbably would notice.
This is just my thought.