Same here, only yom kipur! sometimes not even that!
Every year before yom kipur, I call my rav (weismandl) and ask if I’m ‘allowed’ to fast. Whenever there’s a tiny thing about my health (eg, preg, nursing, after my operation,etc) he doesn’t let, he says just to take ‘shiurim’.
I also was sure everybody does the same, but I see I made a mistake!
About those comments of ‘you’re so lucky you can eat’, in the beginning, I was furious, but I got used to that, i don’t even hear it anymore! As they say in yiddish “az m’geveint zich in tsures, lebt men in freidn”. I know it’s not such a bad ‘tsure, but you get what I’m talking about, hopefully!