Tiawd – Thanks for bringing up this very important topic(thanks also to Reb Eliezer and Avira for backing you up).
You are correct in what you say.
In fact, I recently heard a leading Lubavitcher scholar, Simon Jacobson, in a shiur he gave in Tanya on the radio, admit that it was not literal (this was maybe around six weeks ago. You might be able to find it on his website). Of course, less knowledgeable and sophisticated Lubavitchers might take issue with that, but they are not on the level of Jacobson.
It is unfortunate that some non-Lubavitchers have been infected by this misunderstanding too. Hey, if they already are taking things from Habad, let them at least get them from a scholar like Jacobson, rather than from an ignoramus.