From every tribe of Israel, you should send to the army.
So you have a choice.
Either you say there were not only chareidim, in which case obviously the non-chareidim were sent.
Or you can say everyone then was chareidim. Which is what I think. If you concede that, then we’ll discuss the army.