this is an interesting question and I have thought it myself several times.
I live in a city where buses are not a ready option. Sometimes I will pass a man walking in extreme weather or in pouring rain and if I do not have a son with me, I painfully refrain from offering a ride. Most of the men would know who I am and that my intentions are good, and we are only talking about blocks, not miles, but I feel uncomfortable – or that he would feel uncomfortable.
either way, if your wife isn’t liking it, i guess that answers that question.