Of course we have to follow the Torah Sheb’al peh. There is a great deal of disagreement among poskim as to what the end result of certain halachos might be.
By the time most people have accepted a chumrah upon themselves, MANY people in their community have likewise done so, and then they ALL tend to forget that it is a self-imposed chumrah. They regard it as the absolute halacha. it is human nature to believe that you are better than someone else, when you do something strict that they do not. It’s kind of the way that strict vegetarians believe themselves to be superior to people who eat meat. The meat eaters are doing absolutely nothing wrong, but the people who are machmir on their own eating habits, feel greatly superior because of it.
It never seems to be about accepting a personal chumrah on oneself. My rov ZT”L was one of the few people who actually did that. He would pasken something as muttar for the person asking the shailah, though we knew he did not do it himself.