less chumros:
You’re thinking seems to be validated by tanach. We seem to have Yehuda consorting with a prostitute, the dor midbar worshiping the Eigel, and Dovid hamelech cheating with a married woman and killing her husband.
The Alter of Slabodka, in Ohr Hatzafun addresses this issue. He says that the Torah, in recounting the failings of tzadikim, exaggerates them. This is because on their level, even minor failings become great sins.
This is evidenced by the story of Reuven. The pasuk relates that Reuven lived with Bilhah, his father’s pilegesh. Chazal tell us that what he actually did was to move his father’s bed from the tent of Bilhah to his mother’s tent. His intention was to stand up for the honor of Leah, that she should not be secondary to a pilegesh.
Nevertheless, the Torah writes that he lived with Bilhah, as this seemingly minor sin was counted as if he had violated his father by living with his wife. Indeed, Chazal say that had Reuven not been inspired by Yehudah to admit his sin, he would have forfeited his share in Olam Habah.
Similarly are all the sins recounted against the Avos. The sin itself was always minor by our standards, yet, is “blown out of proportion” in the recounting, in order to show the true seriousness in light of their lofty levels.
Often, by reading the midrashim on the incident, one can discover the actual story. However, even if it still seems serious, it is axiomatic that anyone who is considered by the Torah and chazal to have been a tzadik, will have only sinned on minor levels.