One of the most basic concepts of Judaism is: ??? ????? ???? What this means is that our only purpose in life is to serve Hashem and everything we do is for that purpose.
That means that if someone feels that as part of his Avodas Hashem, he needs to have a “secular” career and/or obtain a “secular” education or get a job instead of learning full-time (or in the case of a girl, marry someone who is working), etc, all of these things can be done l’sheim Shamayim and be part of his Avodas Hashem.
You mention that you feel that most people are either into secular pursuits or into Avodas Hashem. It sounds like you haven’t really encountered people who are pursuing secular pursuits as part of their Avodas Hashem.
I don’t know who the people you know are and why you haven’t encountered this. I am guessing that part of it might stem from the fact that l’maaseh, most people who are truly into Avodas Hashem nowadays are either not interested in or don’t need the secular pursuits. Or even more likely, perhaps most people find that in the world today, pursuing a secular career while being completely focused on Avodas Hashem is practically impossible. Most of the people who talk about “Torah and Worldly Pursuits” are not talking about living a life of Avodas Hashem and serving Hashem in different ways. They are actually talking about living a double life (whether or not they aware of this).
Rav Hutner, zatsal differentiatied between living a double life and living a broad life (one in which the entire focus is on serving Hashem but this is done through various means such as a “secular” career).
However, my guess is that in today’s world, it is not realistic for most people, and most people do find that they have to choose.