They base themselves on David hamelech who killed his yetser through fasts. So it is, in his case at least , due to his actions.
But the question remains – what does this have to do with INFALLIBILTY ?
Note that Menachem agreed that there is no source for INFALLIBILITY .
And – if the navi [!] Shmuel could be mistaken about Yishay’s sons – ha’adam yir’eh le’einayim , then why can we not be mistaken about someone who lives with us ?
.