“I don’t see why that should be the case if chumros are used correctly (kept for the right reasons and not imposed upon others).”
The way to know if someone’s personal chumrah is being kept correctly and for the right reason, is if that person is not making a big display of doing his chumrah. If everyone knows about it, then maybe his reason is a gaivehdig one, so that everyone will look at what a tzassik he is (and mind you, this might even be subconscious on his part). Certainly as soon as someone decides other people who do NOT follow the same chumrah are not as frum as he is, he is already in the wrong.