This issue is actually recorded in the midrash (Mekhilta, Ex. 20:7) as a difference of opinion between Beis Hillel and Beis Shammai (Beis Shammai behaved as you suggested, though with verbalization as others have noted).
The Mishna Berurah (250:2) clarifies the dispute and suggests that Beis Hillel wasn’t entirely opposed to the position of Beis Shammai.