Grey matter asked an important question, but the answer, I believe, is quite simple.
We are instructed to have emunas chachomim, and that means to believe in those who seem to be chachomim/tzaddikim until we know otherwise. Yes, sometimes we make mistakes, and sometimes (rarely) someone who was a chochom/tzaddik can go off, but until we KNOW that we have to deal with the chazokoh that he hasn’t. If that wasn’t the case we would have total anarchy with everybody doing “hayoshor b’einov”.