It is offensive because people who are developmentally delayed (and their families) find it offensive. If deaf people found the term “deaf” offensive then it would too would be offensive.
I don’t know why people have trouble with this. similarly If native Americans find the term “redskin” offensive then it is offensive it doesnt matter how long the term has been used. You don’t have to care you are free to use offensive terms such as retarded and redskins, but we dont get to dictate what should and shouldn’t offend people. If they are offended then by definition it is offensive, whether or not the person using the term meant nay offense.