I did not use the LH argument, however to answer
There is a differnce between LH of someone who is not relgious which is between him and hashem , That is for hashem to decide. If you want to say Kol Bnei Yisroel Averim Zeh L’Zeh. fine but that is more indefinate and indirect than anything
However someone who cheats or is an absuer is a menace to society. Even I dont know the person directly. i might know of someone who does know them and they would need to know to stay away from that person