Depends why you dont eat chalav stam.
If it is becasue of concern for tarfus (either becasue of litteral concern for treifah or chalv akum) then you shouldnt.
IF you hold it is mutar but follow a personal chumra along the lines of “baal nefesh” then you can decide where to draw the line. Be makpid on keilim completly, only when ben yomo, or never.
If you arent sure why you avoid it, you can eat.