mordern – I don’t see the Olam walking around in Frocks like Roshei Yeshiva.
So if you dress like a penguin, you’ll be like one on the inside too.
Which is DY’s heter, because wearing H&J makes him feel like he is cool (cold temp.) like a penguin 🙂
I personally would not wear H&J in the street, only because it is a chumrah D’asi Lidei Kulah, as if you wear H&J in the street, you are mechuyav to wear on for davening (and vice versa). Offen Mishna Berurah “K’derech Sheholchin B’rechov”. If you only wear it for davening, then it isn’t a chiyuv so you can still daven without it if you forget it. (Ask your LOR for actual Halacha).