Joseph, I personally disagree that most are tzadikim. The men I know in kollel range from the people I know with the worst middos to tzadikim. They are no better or worse than the general frum population, they just get their salary from the rest of us.
I have no trouble supporting really special learners. I have a problem supporting people who either 1) are not particularly great learners and should not be learning full time or 2) people who dont internalize Torah. Its great to learn well, but when you have zero middos it shows me that all that learning is doing nothing for them. They dont understand a word of Torah really if their middos are that bad.
And yes, I get to make a judgement because they are asking for MY money.