Why would you think that its permitted. Just because something is with your spouse doesn’tmean its muttar. You cant eat treif with your spouse. You cant be mechallel shabbos with your spouse.
Every frum Jew knows that mixed swimming is completely and utterly assur. I don’t know of anyone who make a hetter for a married couple. Why dont you show me where halacha says that its permitted?