Bava Basra 57b.
I don’t know that it’s necessarily “Assur”. I do know that, unless there’s no other choice, you’re called a Rasha for doing it. Maybe if the only gym in town that you can afford is mixed then it would be Muttar (if one was perfectly sure they could avoid looking at the women there). But it’s certainly not a good idea.