Barry
Hanistarim L’Hashem Elokeinu. We cant tell what is internal. People decide to wear or not wear a black hat knowing full well which gruor they are therby self-identifying with. There is nothing wrong with judging people as belonging to a group that they self-identify wih.
Of course we should eb careful to be openminded enough to reevaluate once you get to know people.
Your example had nothing to do with a black hat, its not like if he wore a kipa seruga and sold treif chickens people woud have expected it. (I hope)