The bottom line is that the VAST VAST majority of Kollel yungerleit are Tzadikim, that we should be aspiring to emulate. (And we should be seeking to boost their ranks — as is they constitute a TINY percentage of Klal Yisroel.)
For one to say otherwise is to be a Baal Loshon Hora and a Baal Sheker, as well as an afront on the Torah itself. And it comes with all the shreklach consequences of someone in that position.