RebbeDebbie, if you are uncomfortable with the type of guests ( divorced or not) you shouldn’t feel any guilt about not inviting them. The atmosphere needs to be on your spiritual level and it’s not a mitzvah to be dragged down spiritually.
We must remember that while the Torah mentioned hacnasos orchim is great, whether in Tanach or in later sources, the women used to dress way more modestly, in certain times and places even their faces were covered. Certainly, the garments used to be looser and longer than the standard today. There’s also less formality between the genders in this generation.
One must try to preserve theirs and their family’s kedusha even at the expense of this mitzvah that can easily turn into an aveirah of tarivus.