The fact is that women are protected as a “minority” under the law and segregating or discriminating against them is illegal, regardless of whatever perfect logical reasons we can give.
The real fact is, that women are not treated the same as blacks in the law, and if you have a good reason, then you are allowed to discriminate. So, for example, it is not illegal for the military to discriminate in terms of what positions a woman can hold in the army, even though it is illegal for them to do the same with blacks even if there was a good reason.