I think that most people who are insistent on buying milk davka labeled “Halav Yisrael” do so out of mystical considerations, as opposed to concrete halakha. The Rambam, Ma’achalos Asuros 3:13, links the prohibition of non-Jewish milk specifically to the risk of non-kosher milk being mixed in; when there is no suspicion of treif milk according to the halacha here, there would be no prohibition. The Rivash and Rashbash, likewise, learn the gemara (Avodah Zarah 35b) as not instituting an across-the-board gezera; they permitted the milk of non-Jews without a Jew present at the milking when it was known in a particular region
that non-kosher animals were not being milked. The Radbaz and Pri Chadash felt the same, and the latter paskens that when it is presumed that the milk being sold by the non-Jews is from kosher animals, based on the fact that non-kosher animals were not
being commercially milked in that region and, in any event, if some non-kosher milk was around it was more expensive than kosher milk (as is normally the case when non-kosher animals are not commercially milked), one may partake of such milk and rely on the presumption that the non-Jews would not conduct in an
forbidden). The Talmud permitted the muryas of non- Jews when wine was more expensive than fish oil.
Even the Hazon Ish holds like Reb Moshe and all the rishonim I quoted above. I don’t see any compelling halakhic reason for “halav yisrael.”