Joseph, YOU (and others) use it as a derogatory term. Originally there was NO term for it. People started calling it chalav akum, despite Rav Moshes psak. Kashrus agencies started calling it chalav stam to differentiate it from actual cholov yisroel which was the term used by those who were promoting their milking being supervised by a jew. The term was then coopted by those who share the sentiments of joseph here and user it as a pejorative term.
In short, everyone uses it as a descriptive term. some choose to use it negatively.