That wasn’t how he said it.
He was saying that the non-frum take their cues from us about what we treat as really important, and then they do only that.
So if a jew in vilna is mezalzel shabbos but still doesn’t do melacha, then the jew in paris does melacha.
Nothing mystical.