YES! You Are WRONG!!!!
From my experience, it means “we are very much trying to cut back and have invited as few people as we can to the dinner, but our Simcha would be lacking without you and we know that you will understand so we are inviting you for dessert/ Simchas Chosson V’ Kallah/ etc”
I enjoy going to these Simchas as I know the Baal Simcha knows I am coming just to join in his Simcha. It’s also so much easier on the schedule as one can come and go quickly without any hard feelings (which, BTW, is another thing Baalie Simcha often have in mind- they don’t want people to feel obligated to spend a whole evening at their Simcha)