48 years ago today the new President took off his hat and it was downhill for the hat business.
Minhag HaMakom (community dress standards) plays a big role in issues like this.
If the Rabbi and community leaders dress one way, following them is probably safest.