Avi K, you can make an argument that there is room in our society for schools or yeshivos where students with superior intellect or who have demonstrated academic excellence can thrive. One can suggest that separate classes for intellectually gifted students are as necessary as specialized programs for students who struggle with learning disabilities. The argument goes all the way back to Bais Hillel and Bais Shammai.
There is no need or possible excuse for schools catering to elite families.
A few years ago there was a video making the rounds of a menahel of a yeshiva in E”Y (Bet Shemesh possibly) who went to consult with Rav Shteinman. The menahel did not want to accept children from a certain family into his yeshiva. Rav Shteinman can be heard clearly telling the menahel that there is no excuse not to accept the children, “Dos is Ga’avah!”
As for your comment regarding Talmidei Chachamim I will respond, though hesitantly, as I’m not acquainted with the gemara you refer to. The non-kollel member of today who’s wearing a blue shirt is hardly ever a true am ha’aretz. While no one will say he has the Torah knowledge of let’s say your local Rav and posseik, he most likely has gone through a decent amount of Shas and poskim himself. Excluding his children from the same opportunities as those of the elite is simply a crime, even if his only claim to fame is that he is Shomer Torah u’Mitzvos and wants to afford his children the best possible Torah education.
And the plural is Amei Ha’aretz. (Or in popular jargon, for those who are not straddling safsalei bais hamedrash 24/7, Ammirratzim.)