I agree that having any “system” can have negative consequences for those outside the system, but overall, despite some silly (maybe that’s an understatement) cultural things, as I said, the kollel system has raised the bar for everyone.
I think if someone’s approach to learning is the same (I know a bunch of people who work but are in the beis medrash with chavrusas or learning themselves for hours a day, and Sundays and work holidays for longer) the kids will be just as well off. Those people would love to learn 12-14 hours every day, but can’t because they need to support their families. Shouldn’t everyone want to?