Avi, the difference an invitation makes is clear: it’s the difference between a public gathering and a private event. His shita is that private events don’t need to be separate, public gatherings do. It’s got nothing to do with how people dress.
It’s not surprising that he says one or two women, on a casual basis, is OK — he says is the case even in a shul, so of course it’s also the case at a shiva house.