Different cultures have different rules as to what is polite when looking at other people. Some cultures look, others don’t. Some cultures count it a challenge when you look for more than a second. If you live in NYC you will encounter all kinds.
So – better not to look, but don’t stare at the ground either. Just keep your eyes ahead where you’re walking (good for not falling in holes, too 🙂 and relax. Shrug your shoulders and tell your friend, “So what? Why should I care?”