The behavior of your employers was unconscionable and inexcusable. I sincerely hope that it never happens to you again. As you can imagine from my user name, I am curious why one would think that a white shirt on a male imparts to him a degree of frumkeit or menschlichkeit. Of course, I don’t think that you personally feel this way, but many people do. It’s another social rule that has nothing to do with our religion but has become one of many litmus tests for social acceptance.