Avirah, reality check! By the frumer Litvishe everybody goes to kollel and some do by the Chassidim. It is not a tiny percentage by any means!
OP, Gemora in Eiruvin says that divrei Torah get preserved only in those who are as cruel to their children as a crow (by giving them substandard food).
Menu