Trying my best – if that were the case, then CS would not be ok at all.
Also, the yotze v’nichnas mashgiach is believed to say that the milk in the restaurant is cholov yisroel, even though he’s not there all the time, did not see the CY milk when it was purchased, and the chef can substitute other milk when the mashgiach is not around.