By “sweaters” do you mean:
1. Solid black (or white, dark blue or grey), either a sweater vest or a cardigan (with buttons) or long-sleeve pullover (usually enough of a V-neck for a tie)-preferably matching or coordinated with pants and jacket
2. Something colorful, with a picture or design on it.
I have frequently seen people (including myself, in a mirror) wearing the former, including under a kapote or beckeshe or suit jacket – and I have never seen Bnei Torah wearing the latter