The biggest issue regarding chumros is that they are often put onto the masses when they should be applied only to the select few. In other words, the average frum person is not cut out to live a life that is as restricting as a major gadol. However, the current trend is to make it that every frum person must adhere to these chumros or else they are viewed as being not frum.
Anyone who does not realize that people are people and that even frum people do not all fit a certain category in terms of ability or desire regarding strictness of halachic observance is being naive.