Perhaps the white shirt should be reserved for Shabbos, Yom Tov and weddings…
This gets critical in the summer in certain cities where it is too hot to wear a jacket or suit.
If you eat chulent during the week, it loses its specialness for Shabbos.
Some people need to wear white shirts for their jobs. Then they need to find another way to make the Shabbos dress special.