Wolf, false dilemma. You can refrain from talking *and* wear a hat and jacket.
I don’t know where you daven, but where I daven (quite a yeshivishe place), the person sans hat and jacket is welcome (although IMO he’s wrong for not conforming to minhag hamakom), but the talker is made to feel quite unwelcome.