The answer is simple. White shirts are part of the uniform of what has become a religion of external appearances. How very sad! What are people concerned with when looking at potential shidduchim? Be honest. To drive my point home I know someone whose regular attire was jeans and t shirts. Certainly nothing close to yeshivish attire. But he is such a huge Talmid chochom that the regular magid shiur has him give the daf yomi shiur when he’s away.
And if someone wears what’s now called business casual? By modern standards that’s a very clean put together look. Wearing a white shirt is not the only way to look well dressed and refined.