Are you talking about chumros for individuals to keep, or about stricter standards to be imposed upon the community?
The down side of chumros is that they can drive many away from keeping Torah and Mitzvos.
Further chumros can create divisions within the larger Jewish community.
The entire tractate of Demai is about the tension between maintaining standards and maintaining the unity of the Jewish people.
Sometimes chumros can take priority over true biblical concerns. I have heard of those who will resort to gezel (theft) to enforce new stringencies in Tznius.