n0, another theory about the past, both re:men and women. Jewish community survived not necessarily because every person was a tzadik gamur. Some were drinking Torah, others were drinking something else. Lots of people were often not on a level. But, in many centuries, these people remained Jewish. In Eretz Israel, they might have even served idols.
So, what used to happen with these people? they stayed within Jewish community and, hopefully, things would get better after a couple of generations. In the times when we are in close contact with other civilizations (now, Germany, Spain, Chanuka), everyone off the derech will assimilate and leave Jewish community forever. This explains current conservatism out of fear of losing members to assimilation.