Every Jewish person is your friend. Also with less “friendly” coversation there might possibly be less chance of slip-ups in the department of lashon hara – I have overheard such conversation in public, with the defence beihg – “It’s O.K.-I’m her friend”!
Be happy all is fine – you don’t have to live up to everyone’s expectations – which may not be suitable or right for you. Don’t feel lonely, you are not alone.