Maybe most men back in the day wore hats. Why take them off at shul? When approaching nobility, nonJews would take off their hats before the royals. By refusing to remove their hats, JewIsh men put Hashem before all. Maybe they brought that back to shul, showing that Hashem is above man.
Also, perhaps taking off a hat would have gotten messy. They could have gotten mixed up, and storage was lacking. They could be big investments, and not something to just toss to the side. Perhaps holding one’s hat would also get in the way of davening, especially when it comes to reading from the siddur.