mik5: Your rav was probably referring to the vast majority of non-frum Jews, who are adjudges tinokos shenishbu. basically, this means they cannot be held responsible for their failure to observe the Torah, and the fact that they don’t follow halacha does not impact their personal status (i.e., they are not ajudged mumarim l’hachis).
The concept of tinok shenisha originally refers to a young Jewish child captured at a young age and held in captivity by non-Jews for an extended period. The influence of being raised without a Torah education and outside a Torah environment is considered an excuse (not a justification) for why the grown up child fails to follow halacha.
In the contemporary context, it refers to all those that have been raised by secularized or non-observant parents, who are excused for their non-observance of halacha because of their being raised in an environment and culture in which Torah-observance is so foreign a concept, that they cannot be expected to have any regard for it.