It was never for “the best” learners but rather for anyone who WANTED the honor of learning in Kollel, as the Rambam describes those who WANT to join Shevet Levi. He does not condition their membership in the Kollel community as having to be the best, but rather having the desire. The idea that only the “best’ should learn in Kollel is a baseless falsehood and it is against the Halchah as expressed by the Rambam which states that anyone who so chooses may learn in Kollel. See also YD Laws of Talmud Torah 246:21 and Shach ad loc. Kollel is a special privilege and status that anyone can go for if they so choose, the Rambam says.
Sitting and learning all day is the ideal. “Talmud Torah kneged kulam.” Chazal say, one word of Torah is higher than an entire lifetime of doing these Mitzvos. Chazal often mention that Toroso Umnoso is the ideal, that we do nothing all day but learn. Nowadays poskim say that w cannot reach that level, but clearly the closer the better. Also, Shulchan Aruch Hilchos Toalmud Torah, in the Shach ad loc, says that nowadays learning all day is the ideal, and that if someone has the ability to do it, he should. The Shach adds that regarding learning all day in general, nowadays we cannot reach our potential in learning the way the Rambam etc. did, since we are not on that level. Therefore, we should learn all day if we can.
The Rambam writes that a “working person” is someone who learns 8 hours a day and works 3 hours. Not works 9am to 5pm.