Let’s keep in mind that for centuries the majority of Jews were not very learned. Poverty in Eastern Euroupe was very very rampant. That meant kids needed to start working at a young age to help support the family. In addition the tools we have today were just not there. Seforim were very hard to come by. One shas in the whole town? The entire population of the yeshiva world combined at the HEIGHT of the yeshiva movement in Lithuania was about 3,000. That is not even half of the just the Mir today.
That said Torah is given to Jews as a gift based on the amount of yegia and effort a person sacrifices for it. It could be that despite all of these difficulties there were a decent amount of people who could learn gemara on a low level and even fewer who we would classify as gedolim.