To be clear… just because it says something in a teshuvah from Rav Moshe does not always mean that was the final word. Rav Moshe felt that due to the gov. supervision there was no problem whatsoever with standard American milk. The reason that the teshuvos state the facts the way they do is because Rav Moshe did not want to eradicate the idea of Cholov Yisroel in America. It is for that reason that he answer the shayla of a school using cholov stam the way he did.
As far as why cholov yisroel spoils faster the answer is simple. The cholov yisroel dairy’s are much more cost conscious that the major cholov stam dairy’s. Because of this they tend to have older trucks and less efficient refrigeration as well as more delivery’s by the same truck in one day. This means that your milk is sitting in a non ideal truck for longer and is sometimes (especially problematic in the summer) sitting outside for some time before the store opens.