It is important to note that this conversation does not apply equally to all kollelim.
Many kollelim (most notably, nearly all of the ones in what are colloquially, [and rather small-mindedly] referred to as “out of town communities”) actually pay a significant salary. While rarely enough to live on without supplementary income, this salary is definitely on par with that of many other jobs.
In my community, the most needy families (which put the greatest burden on the community) are not the kollel families, but families of unsuccessful businessmen.
What I have observed firsthand, and heard from longstanding members of various communities which are fortunate enough to host kollelim, is that a kollel is a tremendous positive force within a community in a myriad of ways. While it is obviously true that the kollel and its members rely heavily on the rest of the community for support, any fair minded individual should recognize that it is a symbiotic relationship.
Finally, I will mention that although I have had this debate many times, I have never heard this complaint from someone who did not have a personal enmity for “black hatters.”