Rambam says that Kollel is for anyone who WANTS the honor of learning in Kollel (Shevet Levi). Also see YD Hilchos Talmud Torah 246:21 and Shach ad loc.
But this is a fig leaf. Only a small minority of Chareidi Yidden in E.Y. are in Kollel. Where are the complaints that they should stop supporting the seculars university education? Roman literature and Humanities yes, and Torah no?