I read an article about this a couple months ago. It seems OP is correct, that there was a levush yehudi. At some point, the goyim outlawed it, and the litvaks stopped wearing it, but the chassidim refused. Ultimately, the chassidim settled with the goyim that they would wear the similar garb of the noblemen, which is what they continue to wear today.
Thus, the chassidim are really more correct about this. However, I have not reverted to their dress, since my rebbeim don’t.