There were indeed great people who didn’t keep zmanin for tefillah. There is actually a halachic explanation for it, albeit not one that the litvaks would go for.
But to say that certain great tzaddikim weren’t great is small minded, because they had a cheshbon, even if we don’t understand/agree.
As R’ Moshe famously said about one tzaddik, “He deserves petch, but there’s nobody in this generation wortht of giving it to him”.