Haifagirl:
It is considered offensive to use “retard” as an insult because such use is hurtful to the developmentally disabled (for whom it was previously used exclusively as a technical name).
It is considered offensive to use “retard” for a developmentally disabled person because it has acquired a negative connotation due to being used as an insult.
When the person using the word in relation to a DD person is a close relative of said person, it is obvious that they do not mean it negatively. But this does not mean that the word is appropriate for general use. In your cases, the people using the word may not even be aware that many consider the word offensive.
(And would even those people use the word for that son or sister in their presence?)