Since you can investigate, it is your responsibility to make sure the money is going to a halachic Tzdedaka.
Why should legitimate Oniyim and causes get short changed because someone is too lazy to do a chakira, an investigation, on every person who sticks out their hand? A Baal Tzedaka means that should not be hefkeirus in alloting funds; you are the Baal of the Tzedaka, with a cheshbon.
In my shul, those who are given the most, are collectors who approach the Rav first with shalom and and brief introduction. The Rav will usually put in a verbal plug for the cause or allow the collector to advocate.
What about collectors who come in teams to the home or close to Shabbos or with props?