The Psak to eat Cholov Yisroel only is based on the Psak that non-CY (what is usually called Cholov Stam) is completely forbidden to consume according to the poskim who insist on CY-only.
Even keilim that are non-CY are assur according to these poskim.