I’ve thought long and hard about it in my day, and I’ve come to this conclusion:
There’s the tznius factor, of course, as others have pointed out, but it’s a misconception that chassidim have singled out women in this respect. Driving is frowned upon, period. It’s perceived to be dangerous physically and spiritually; if you’d think twice before driving somewhere, you’d think ten times before taking a taxi there. A driving boy is considered like a working boy; he can forget about a good shidduch. Shaina yidden, ????????? ????????? (sorry, defies translation) and many rebbishe einiklach, wouldn’t be caught dead driving. I suspect it would be assered for everyone, if possible, but that would be a ????? ???? ?????? ???? ????? ??. So working men drive because they have to, and many non-working married men who need a car for their ego drive because it’s already acceptable for their age and station. Still, the ideal is to not drive, for everyone, not just women.