The Ben Ish Hai says that even if one davens at home one should wear a shirt as one would not walk in the street without one. Rav Mordechai Eliahu learns that if someone would walk in the street without a hat a jacket he need not wear them for davening.
As for what was, if you look at old pictures you will see that a man’s hat signified his station in life. Workers wore caps. Middle class men wore hats that matched (and came with) their suits.