Okay so on the other hand, who is this guy in my driveway? Better to ask him to move than call the police because a suspicious person is waiting in my driveway.
How is the homeowner supposed to know that the other person isn’t up to something? Even someone who looks like a good Jew might be trouble. Maybe it’s a privacy issue too. Some people leave their curtains open. Or are simply walking inside in their pjs. It could be uncomfortable to have a stranger basically waiting around in your paved front yard.