lilmod: Please read Rav Shlomo Zalman’s reponse in its original letter. (It’s reprinted in the TE.) Your understanding of it is mistaken. He holds it is only to be done b’dieved in the sense that if it is unavoidable. i.e. a crowded street with women. Not to b’davka do it.
Additionally, the vast majority of poskim, both historic and contemporary, do not pasken to be meikel on this issue. The Shulchan Aruch rules it assur. If any posek would hold differently than the S”A he would openly state as such. The absence of a ruling mattiring what the S”A assured, is a clear indication that he holds it assur. The Gedolim in the days of the Shulchan Aruch and shortly thereafter have agreed to accept the psakim of the mechaber and the Rema as authoritative. The Shach writes that one cannot even claim “kim li” against a psak of the Shulchan Aruch. This is akin to accepting someone as your “Rebbi”, where you follow his psakim. This is the same thing that happened when, let’s say, Klal Yisroel decided that the period of Chazal has ended after the 7th generation of Amorayim (Mar Zutra, Mar bar Rav Ashi, etc), and nobody from here on in can add to the Gemora. There was no “halachah l’Moshe m’Sinai” that told us that the Gemora was sealed; it was the accepted reality told to us by our Gedolim. The same thing applies to accepting the Shulchan Aruch and Rema.
Furthermore, one cannot be meikel on an issue that S”A assurs simply because there is an opinion out there, even a valid one, that isn’t from his rebbi or posek, being meikel on an issue. Even if someone else will be insulted that you didn’t latch on to the kulah and you instead followed the halacha as the Shulchan Aruch paskens and as your rebbeim pasken.