Mammele, Rashi disagrees with that chumra (Yevamot 88a d”h v’amar). It certainly is no way to keep a community intact, not to mention families. In fact, Rav Moshe ate at a simcha given by a baal bayit. when asked he said “A single witness is believed for prohibitions. the Halacha allows me to eat. On the other hand, embarrassing someone is a Torah prohibition” (see Shailot Yaavetz 2,15). In fact, there are some opinions that a person who does not go by that rule is a heretic – he disputes Chazal (Pitchei Teshuva YD 116,10). The Ran says that even if the food is not kosher he does not get any aveira. However, if he decides that he is smarter than Chazal and he slips he is responsible.
Of course, some people feel the need to say “Mirror, mirror on the wall. Who is the frummest of them all?”