“none of the Poskim read it Kipshuto.”
Aha! There’s the answer in your own words! All the poskim say something and them some yodle comes along the highway and matirs what is assur, “against everyone else” as you so elequently put it?
See, after thinking about it a little bit you figured out the answer yourself! 🙂