It doesnt bother me when people wear hats and jackets in 90 degree weather, I just do think it looks stupid.
If taking off your hat and jacket (especially in a situation that is not relating to davening or learning) makes you throw off halacha, then your Judaism is only suit deep. To me that is very sad.
I laugh at the fact that non-Jewish clothing is the symbol of “Jewish garb” – look back at WHY the chasidim started wearing the clothing of Polish royalty and you may enjoy the joke with me.