OK, I thought it was clearer than that, but I’m happy to spell it out. A husband and wife are at times prohibited from sharing a bed, since it is kiruv l’arayyos (KLA), even if both sleep wearing full nightclothes. At times, but not at all times (I hope that is clear enough). If you hold that mixed swimming carries an intrinsic issur (of KLA), then the same issue applies.
IOW, exactly what you have been saying all along (that at times husband and wife may swim together).