you are right brisker that is proper derech eretz
on the other hand
sometimes these people are makpid to fulfill the Mitzvah of eating on Shabbos before chatzoz
sometimes they are diabetic
sometimes they have family and guests waiting for them at home
sometimes they are plain ravenous after not having eaten all morning
sometimes the baalai simcha should be more considerate and go directly to Kiddush and shmooze there