Shopping put it beautifully.
It’s not about looking down on people who wear non-white shirts. I will tell you, behind internet anonymity, that I really don’t think shirt color should matter and I wish it didn’t, but the reality is that it does socially. Nobody so far here has asserted that it matters halachically, so stop pretending we are saying that. Where I learn and where I live, some people wear white shirts and some people wear colored. I wear white, not because I think it’s a halachic issue, but because wearing colored would cause people to associate me with the leftern hemisphere of the community.
Now, pro-colored folks, go ahead and attack me for being mean for not wanting to be associated with the religious left, because we all know that’s what this is really about. That, my friends, was exactly Shopping613’s point.