to the OP:
In yeshiva I had a friend who did the same thing. He asked a shailah to a YU rosh yeshiva (who wears a hat) if he could wear his hat when he went home, since he lived in an MO community. The answer was that he could do it, as long as he didn’t show off and it didn’t make a big scene.