It states ולא תלכו בחוקת הגוי – “You shouldn’t walk in the ways and customs of the nations.” The Rambam states in Hilchos Avodah Zarah that you shouldn’t follow the practices of the goyim. Not because it’s a certain practice that is wrong, but just because we should be different from the gentiles in the way we enjoy, in the way we behave, and even in the way we clothe ourselves. That’s what the Rambam says. Otherwise, if you participate in the things that goyim do, it’s not just that you’re sitting in a baseball stadium. The real problem is that your mind and your dei’os, which means your character traits, will begin imitating the nations with whom you are associating.
It’s important to be apart from wicked Jews too. (Not only wicked Jews, even Orthodox Jews who are fools. There are plenty of fools by the Orthodox too; oh yes! You have to be careful with whom you sit with in a beis haknesses. A fool sitting next to you can be speaking loshon harah or leitzanus during the davening. You have to be careful.)
The Rambam says דרך ברייתו של אדם להיות נמשך בדעותיו ובדרכיו אחר רעיו וחבריו – “A person is drawn in the way he thinks and the way he behaves according to with whom he associates. Even the Orthodox are included in that.