When dealing with non frum relatives, you must have boundaries. But you must know that wherever you draw those lines, there will be pressure against them. They will never be happy with them. So you might as well draw those lines where you are comfortable. No sense both of you being miserable about it.
I think kids can handle that grandma and grandpa don’t keep Shabbos. That is common enough (nebach) nowadays. But the dei’os kozvos, the wrong ideas coming out of their mouths, might sink into their heads softly, after hearing it enough times. They’ll go on and begin to wonder themselves later on. As Goebbals, ym’s, said many times, repeat a lie often anough, and they’ll start to believe it.