Newposter,
Yes, technically you are correct. Cholov Yisroel is an actual halacha.
However, what Cholov Yisroel means today is not quite the same as it meant when first enacted.
The point of Cholov Yisroel is to make sure that the milk you are getting is genuine cow’s milk (or milk from another kosher animal). It is not a decree (such as bishul akum, for example) against intermarriage.
Since the main concern of ChY is whether or not the milk is genuine cow’s milk, then if you can be sure that the milk is kosher, then there is no need to have it watched by a Jew.
The question then boils down to is the government a good enough shomer to make sure the milk is cow’s milk. Those that hold that it’s okay to eat Cholov Stam maintain that the government puts enough mechanisms in place to ensure the milk is genuine cow’s milk. Those that only eat ChY (as it’s termed today) apparently maintain that that’s not the case. In that sense, ChY is a chumra.
Nonetheless, everyone agrees that if you travel somewhere where governmental authority cannot guarantee that the milk is kosher (think Vietnam, Laos, Zimbabwe, etc.), then you must only drink milk watched by a Jew as the halacha of ChY dictates.
The Wolf