Growing up, we always gave candy to trick-or-treaters. We still do, although there are less and less non-Jews that go trick-or-treating in my neighborhood every year.
In my opinion, there is nothing wrong with not opening your door to them. Plenty of people don’t. But it would be very wrong to do anything that would show scorn or contempt.