Forget the person. Is the hat a yiray shamayim?
If someone would tell me that wearing a black hat (or a purple cardigan) improves their yiras shamayim, I would tell them to make sure to own one for each day of the week and a super nice one for shabbos. Just wearing it because it is an identifier, is not necessarily a bad thing, but other choose other identifiers to also connote “ben torah”. Lastly, what does it mean to a person when they make the statement that they belong to a certain group? Belonging to a certain group, does it improve your yiras shamayim, or your shidduch prospects?