There are actually two separate points here; what the best response is, and whether or not the OP’s response was acceptable.
The easier one is the second. It clearly wasn’t. You may not like Joseph’s analogy, and you may choose to ignore him, but that doesn’t make his point wrong. He, and DaMoshe, and apushatayid, and 5ish, and CA are correct.
As far as the correct response, I don’t see how you can ignore the affront to the Torah. How to show disagreement with the sentiment expressed is hard to say, especially not knowing the co-worker or the relationship.