When I was growing up, the concept of white collar worker vs. blue collar work was prevalent. B’Kitzur; white collar workers were professionals; doctors, lawyers, accountants, teachers, etc… whereas blue collar workers were manual workers, factory employess, etc… I am not sure the origin but I would surmise that blue shirts tended to hide the sweat of the employees.
But perhaps from a different perspective; white shirts and black slacks/pants tend to be the “lowest common denominator” in terms of color — lacking color means one does not concern themselves with matching, etc… and therefore little time is wasted when it comes to clothing (in the Yeshiva, Chassidic world).
I am not necessarily disagreeing with colored shirts (I wear them), I simply am trying to find the underlying foundation of a strict white shirt “policy” in some circles/circumstances.