You have a weird pshat into who qualifies as being part of the Jewish nation. Yes, Rav Saadya Gaon is saying that only the Torah makes us Jewish but the same Torah says that someone who is born of a Jewish mother is Jewish. Pshatus is that secular Jews are part of am yisrael.
And I just want to clarify from above that many Chassidim learn properly and don’t miss zman tefilah. But a lot don’t. And I don’t understand why when people compare MO to Chasidish that these variables are left out. Is a frum serious YU boy who keeps halacha but may be a bit more exposed to secular culture worse than a Chasid who doesn’t learn properly and davens at 11?