According to the story you relate, Rav Soloveichik was telling the man not to walk around on shabbos in his shirt sleeves because he was leaving his hat and jacket in shul. (The practice of wearing a broad brimmed fedora without a jacket is a relatively recent shtus.
charliehall: You can write a much longer list than that, including many rabbonim who lived after the fedora came into style. We can start with Rabbi Eliezer Silver.