No, this is not some loshon horo if you expected that.
Avoda Zara 55-57 has a couple of cases when halocha is more lenient for Bnei Torah but stricter for others in order to influence behaviors.
That is, a Ben Torah can eat turmus cooked by goyim while R Yochanan osers it for a town of amei haaretz who do other weird stuff, like marry their daughters to incomplete gerim. In the other case, Rava tries to justify his leniency in Mehuzah (where rich merchants lived) – and gets a response that they are not – comparing with yeshivish Nehardeah …
So, a couple of things that strike me:
1) in our times, those who consider themselves “bnei Torah” try to out-do each other in chumros. Eating turmus without a heksher? please
2) the term itself seems to mean simply people who know and follow halochos well. No implication that they exclusively sit in yeshiva.