It’s funny you should ask….years ago when I was more idealistic,a shidduchim was redt to me and after inquiring,I found out that this boy wore a blue shirt under his suit on shabbos. So I rejected the idea. But you see,fate works in funny ways,because then I got the same exact suggestion from someone else,a real chashuv man. When I told him I already checked and this guy was not for me,he said that I should go out and try it because this person is very ehrlich and a good catch. I really was not so into this whole idea….but it’s good I listened,because he is my husband and although he wears a colored shirt,he is a very decent,upstanding,frummer then many “white shirts” and lots of other nice things that I don’t need to publicize!
So,no,colored shirts shouldn’t matter.
But I know that unfortunately,some do judge by the clothes a person wears.