@Decency,
the fact is, and always was, that people see Jews as a group. How “few” are “few”? You can count yourself by number of people here who protest masks; by number of announcements of levayas and requests for tehilim; by statistics coming of Israeli towns.
Those responsible are wrong even if the whole world will be doing the same thing.
Maybe those without masks should use Rambam’s advice – if you are really into an aveira, go far away, and do it there. He does recommend dressing up in despicable black, but maybe at least take the hats off.