Sam first of all I think mens and womens shirts are different in any case. This has been a minhag for years and years dont know with which specific livushim though but possibly with shirts. The burden is on you to prove why continuing to wear clothes you have been wearing for forever fall under lo silbash. Its not mistaber bichlal. I hope you have a source because to just say its a serious safeik is a very controversial way of saying things and thinking. Did you ever even bounce it off some mavinim before formulating your opinion or you just relied on yourself? Because if not I think saying something like “I honestly think” is a bit too presumptious.
Menu