I think most of the time, those who look down on others are those who eat (or drink) Chalov Stam (not that everyone who eats Chalov Stam, looks down on others, but most of those who do look down on others are those who eat Cholov Stam as opposed to those who eat only Cholov Yisroael), look down on those who don’t, with an attitude that “You don’t think I’m kosher, you are just too fanatical.
I can still keep kosher without being a ‘fanatic’, like you.”