For there to be Kavod HaTorah those who take on the status of “full time learners” must show at least the same dilligence as those who go into law or medicine.
While some of our more prominent doctors may have slacked off during high school or undergraduate study, it is very unlikely they “slid” thru medical school.
Those who learn in Kollel full time do so only “during the pleasure of thier wives”. At any time the wife can take out the kesubah and read it to her husband and make him go off to work.
Even if someone has a large trust fund and wants to be in Kollel, but also want to be able to “slack off”, the Rosh Kollel needs to deny him full time status and insist he do something else at least part time, so as not to dilute the “profession” of Torah.