They feel that he holds himself as better than them because he wears a hat.
He isn’t? :p
In all seriousness, a baseball cap or Skullcap (or porkpie, or football helmet, etc.) should suffice for the Halachic aspect, even if you hold the Mishna Berurah means that a hat is actually required (which is not the pashut reading).
The King’s representative always dresses as befits his station. Whether in the palace or in the market.
Even when among fools who scoff at The King.
As Feivel correctly points out, nowadays, a Black Fedora and dark colored suit jacket (not a Blazer, which you never see in a yeshiva) are team identification. People want to show they are part of the “Yeshivish” team, so they wear the team colors.