Brown vs BOE has been applied to much more than just schooling. And gender segregation is generally treated very similarly to racial segregation. The courts might be a little more lenient but technically they should legally be the same.
I’m guessing you don’t know very much constitutional law. This is completely incorrect.
For further reading, I suggest you read Brown v. Board of Ed. and Craig v. Boren.