It is probably a problem. Physical fitness classes often involve contact, and usually involve immodest clothes. It would probably be best to find a single-gender class (which is typically easier for women then men, since more non-Jewish women are also concerned with modesty than, relatively, non-Jewish men).
Perhaps there should be more of an effort for frum-sponsored single-gender health clubs.