Oomis, look at the Targum by Yosef. They wouldn’t eat a meal together with foreigners since they would be eating what they worship. It always amazed me that they had such religious tollerance.
Anyhow, the Medrash is plain to see, in Rabbah and Tanchuma.