Joseph,
The fact that it is only a small % (given that it has any effect of a Minhag) does not matter, a Minhag has to be accepted by all. If there is one Bes Din in town that doesn’t accept it, it has no effect as a Minhag. Asides that it is not within the boundaries of any Minhag.
What it does do is that if the custom (it used to be in Israel at one point) that only a man drives then it becomes somewhat of a לא תלבש issue. The same way you understand that a woman should not be driving an 18 Wheeler, although there is no specific issur..