If one accepts R’ Vosner’s take that it’s a gross violation of tznius, the application, even if one could hypothesize that it would change the specific psak, cannot be called irrational, illogical, or be called dishonest to present as tznius.
So, no matter how you try to soften it, you are still essentially saying it about R’ Vosner.
Although I believe R’ Vosner may have the only written teshuvah against it, it should be pointed out that each chassidus has its own poskim, and they aren’t beholden to R’ Vosner. The fact that several are against women driving tells me that his opinion is by no means that of a daas yochid.