Right; it’s easier to have a false sense of unity when one is not interested in the subject matter. Conservative and reform jews have no problem getting along with Christians, atheists, and buddhists, because they don’t consider these beliefs to be anathema. It is not due to any love, but rather due to a lack of love of and conviction in Torah and Hashem.
We respect any person because of kovod habrios, he is a tzelem elokim regardless of belief. Whether or not you want to be around a specific person will depend on that person’s influence and potential to cause harm to one’s self or family.