PBA: I think many do. Many consider them good friends and people to talk, hang out, or discuss life and religion with. There’s just a knowledge that they’re not “Frum”. Your question is that a woman can wear pants and still avoid the “not-Ftum” moniker. What’s the Chiluk? Probably that you can trust the Kashrus of a woman who wears pants but not a M’challel Shabbos. That’s an important distinction.
The second half of your post is a good point.
As regards the first half, I’m also friends with non-frum people, and non-jewish people, and religious people of other religions, etc. I think we’re talking here about who you consider part of your religious community.
And I think there is a bigger difference going on here, than just “these are the kitchens I eat in and these are the ones I don’t”