Sam2,
We might think that Kavod Hatzibbur or Kavod Hatorah etc. dictates that we always wear nice clothes (white shirts) to represent that. OC 53:18 tells us that’s not true.
You’re misunderstanding the S.A. . See M.B., and specifically what he says in the name of the Pri Chadash.