So much wisdom has been posted in just a few comments. My father O”H used to tell me that the biggest danger in Yiddishkeit is when people make the tafeil the ikkar and the ikkar is made tafeil.
When the ikkar of halacha says something is muttar, and then someone comes along and decides to be machmir on something related to that halacha, something that there is no real halachic reason to do (like not wearing colored shirts), that is making something basically rather trivial into a be-all and end-all, which by itself actually can result in trivializing the very important concept that a Ben Torah should dress with derech eretz. Does it REALLY matter to Hashem if the shirt is white, as long as it is clean, neat, pressed and worn by someone who has the right hashkafa towards learning and mktzvos? (Don’t bother to answer, because if you believe the answer is “Yes,” you have proved my point to me).
Re: being machmir out of doubt – we always must ask a shailah when there is a doubt. I mentioned on another thread quite a while ago, that something happened and I was faced with a halachic issue. I started to decide to simply be machmir on myself about it, but then realized I shoukld still ask my rov the shailah. Surprisingly, his p’sak was nowhere in the chumrah ballpark in which I would have put myself, and when I told him that I would have been way more strict about the inyan, he replied, “Anyone can be machmir. It takes someone who REALLY knows the halacha to tell when it is muttar.”