Once your RAV told you not to, what is the point in asking the question? You already have your answer.
Had you NOT asked your RAV I would have suggested that you ask the question to a RAV involved in kiruv. Their answer might have been different. Their suggestion might have been to ask where he lives so that you could set him up with someone in his vicinity so that he can truly enjoy the Shabbos experience without driving there. Or he might have told you since he will drive regardless of whether he comes to you or not you are not making him drive so it is ok to invite him. And the kesher he makes with you might be a catalist for him to become shomer torah and mitzvos.