Regarding inviting non-Jews on yom tov:
it was explained to me by a rav that cooking for them is assur m’d’orisa. Inviting them is assur m’d’rabanan. Why? because even if you are only serving sandwiches, what are you going to do if they ask for a cup of tea? And regarding non-Jews who are in the process of converting, it is still not allowed without certain items being done differently (I don’t recall the exact details).
Regarding non-Jewish or intermarried relatives, the most important thing is to always remain civil and pleasant to EVERYONE. Also, while it may be appropraite at some times to no longer have contact with certain people, ba’alei teshuva may have kibud av considerations(like if the sibling is intermarried and it is a big deal to the non-frum parents that they be invited for Thanksgiving, or something like that). Everyone should use common sense, ask shailos, and be civil and pleasant to all.