When traveling in some areas with few frum (strictly Orthodox) Jews (and even sometimes in some very frum areas) I have several times been harrassed by some who were very against the very idea of a Jew wearing a black hat and following Torah.
Some of the worst ones, identified themselves as Jews.
So not looking too much at others is simply my way of trying not to look like I am trying to start any trouble with a stranger who for all I know, might be looking for the slightest excuse, to get into some kind of conflict with one of those ‘backwards fairytale believing, religious fools’ as I have been called, more then once.
I don’t mean to be unfriendly or standoffish, but you just never know who is who untill you are properly introduced, and you do start to get to know them.
May G-d show you his guidence and his plan for your life in such a way that you can embrace it and enjoy it, and grant you peace of mind (which I believe is the single greatest blessing anyone can ever have.).