This is the Shita of Rabbeinu Yeruchem. Rashi held that the Gartel is for Libo Ro’eh although he wore one anyhow. When Rashi once recited Krias Shema without a gartel he explained to his Talmid that our clothing doesn’t really need it. This is in the Shiltei Hagiborim on the Mordechai in Shabbos third Perek (IIRC).
However, we do find that the Medrash, which Rashi brings, on Hasachas Elokim Anochi uses the term His’azer for Tefilla. It seems like it is more than just an expression of some Hachana. The Bach discusses this and mentions that everyone wears one.
It probably went out the window when at one stage modern clothing came in. It is certainly not something Chasidim started. It is something they didn’t drop.