What you would wear to a simcha?!?!?! No need to create a tower of chumras, Rav Moshe Feinstein advises against this when he gives a teshuva on whether one is allowed to put ketchup on chicken on shabbos (which kinda has to do with this…).
Assuming you do not wear denim, sweatpants, or work out pants in public on a typical day when you aren’t working out, you have nothing to worry about.
I am guessing that you are a baal teshuva.