Sounds like a minhag that can spare some hurt feelings. Prevents the situation of Reuven Lo-Gebrokts turning down an invite from Shimon Gebrokts, but accepting an invite from Levi Kol-haChumros, hurting Reuven’s feelings in the process.
We personally do invite and get invited for Pesach meals. If we feel that questions need to be asked, we ask them openly.