Everyone has nisyonos. A well meant Bracha is a beautiful thing. Most people with these struggles I know would appreciate the Bracha rather than feel badly because of it. They are aware of the Nisayon, and appreciate the thoughts of other people.
The opposite scenario- Not giving the Bracha to such a person- They may feel just as bad that you are not thinking about them! Imagine an older single going up to a Chosson/ Kallah to wish Mazal Tov and not getting anything back, just thanks etc. Wouln’t they feel bad they aren’t being thought of?