We’ve done it a few times, without any issues, and they all had a great time. A very lovely coworker/friend of mine (who happens to be black), came to a simcha and I was delighted to have the opportunity to dance with her!
We originally were going to have a mixed table to goishe coworkers, but not enough came to fill a separate table. We explained what the seating would be like, and they had no problems. Too bad non-frum Jews can’t be as accommodating.