Chalav Yisroel is a halacha mefureshes- we are afraid that goyim will adulterate their milk with treif. The Aruch Hashulchan (usually not known for being a machmir) brings down a few horror stories from people who were meikel on butter and ended up eating treif.
Reb Moshe Feinstein ZT”L came up with a chiddush that in a society like america, when there is strong government oversight, nobody would think of adulterating milk, and it is therefore as good as Cholov Yisroel. This is a big chiddush, and many do not hold from it or try to be machmir on chalav yisroel. One thing we have been reminded of during this financial catastrophe is that companies will do anything to improve the bottom line, and that underfunded government agencies are no match for savvy businessmen (Madoff Securities, Satyam, all the companies caught with melamine-laced food products, etc.) The “fear factor” of government is somewhat limited when compared to the fear of the shareholders. Companies can and will cheat however they can to improve the bottom line. That said, milk is a basic staple, and the resulting scandal from being caught adulterating it would bankrupt a company. For other products, it’s not so simple.
I have personally found that chalav yisroel spoils much faster than cholov stam, and drinking spoiled chalav yisroel is just not that good for my system. “Chamira sakanta meissura”. So I’m usually makpid on chalav yisroel, but if there’s no chalav stam I’ll drink it 🙂