Honestly I don’t understand what the question is. If YOU live in a Yeshivish community and your husband goes to that Yeshiva aren’t you obligated to ask the Rosh yeshiva and abide by the rules and customs that he sets for his yeshiva olam?
In all honesty if your husband is part of THAT yeshiva, you should keep to the rules of THAT yeshiva. It might be better to meet the other women for lunch or brunch or tea Shabbos afternoon when the men go to Mincha, if the Rosh prefers that couples don’t mingle then that is what you follow. If on the other hand the Rosh encourages hachnasos orchim, then that is what you follow.