Being that your question seems more a Hashkafa question than Halacha I think it is OK for me to give a possible answer.
We see that the Halacha follows Beit Hillel over Beit Shami even though Beit Shami had the greater scholars. The reason is because Beit Hillel had more Talmidim and we go after the Rov.
Another point is that sof kol sof the final outcome is min hashamiyim. Even though many big poskim came out against Chalav Stam, in the end we see that Chalav Stam is generally excepted at least on a Bidaeved level (such as materna for a baby).
Perhaps the Sefer, “In Search of Torah Wisdom” by Rabbi Yisroel Miller might answer your question.
Hope this helps