It is not always about quantity… 24000 early students of r Akiva were less successful than the later 5.
I heard a similar story about r Schwab refusing to give kibudim to non shomer shabbos people, majority of the congregation left, but modern Baltimore grew out of remaining small shul.