Perhaps the difference is one (milk) has a gezeira from Chazal while the other (other food) doesn’t. Just saying, since you asked for the “difference”.
Trying my best – if that were the case, then CS would not be ok at all.
Actually Reb Moshe holds that CY IS a gezeira, not like the Pri Chadash. However he holds that the USDA supervision fulfils the requirements of the gezeira and is equivalent to CY, read the teshuvos in IM. Rav Gornish follows the Chassidishe Poskim who disagree with Reb Moshe, and is NO reflection on what Reb Moshe himself held.