I dont think those who only eat Chalav Yisroel are jealous of those who dont , nor do I think those who eat Chalav Stam feel guilty.
However I do think those who eat chalav Yisroel sort of have this holier than Thou attitude over those who dont. If It has an O-U on it, its perfectly fine. And Ben and Jerry’s is 10000% better than Mehadrin. Quality is important. there is no Mitzvah to eat lousy Ice Cream