Gaon
“However I recall, Rav Moshe does quote the Chasam Sofer and argues on him.”
R’ Moshe is saying his chidush even in chasam Sofer. The Gist of R’ Moshe’s Chidush is that it is sort of an “Anan Sahadi” we are so sure that there is no treif milk mixed in that it is as if we watched it milked.
In his first teshuva on the subject YD 1:47 he begins by saying his heter does not rely on the Pri chadash rather that “ידיעה ברורה הוא כריאה ממש”
Though your main point “the point was that the ones not drinking in many cases are not just doing it as a mere Chumra.”Is of course completely correct.