Should you make an illegal driveway? No. Of course not. It is unlawful and likely assur.
The question here is how do you deal with someone who already did make a driveway, and uses it?
The answer, to me, is find another spot. For sure in terms of the right thing to do. But even if it is a question of “rights” I still think a person should find a different spot unless desperate. You are entitled to disagree, and I’m entitled, as usual, to think that the fact that I’m on the other side of you on an issue, tends to show that I’m right.