AAQ – I think R Hirsh means that Yitzhak needed to observe people who think differently, not just those who were already talmidei Avraham.
The thousands of yokels who walked in the door for the free food were not yet “Talmidei Avraham”, and most probably didn’t end up ‘going all the way’. Probably more like a Chabad House, and less like a Yeshivah.