Kaddish is a zechus for the neshoma of the niftar and a nechoma (comfort) for the child, sibling, spouse or (R”L) parent who is saying the kaddish. It is never, ever a tircha to listen to and respond to someone saying the mourner’s kaddish. As a matter of fact, I wish those who talk during kaddish would be more aware not only of the halachic problem, but also how it may hurt those saying the kaddish who may perceive disrespect and uncaring toward the niftar.
Even those who have never been frum tend to highly value the saying of kaddish. At work we’d regularly have non-frum people come in to our minyan on yortzeits, put on a yarmulka, and say the kaddish for their relatives. Sometimes they couldn’t even read Hebrew or say the “ches” properly, but they’d do their best reading the English transliterated version they had. My wife paid a yeshiva a few hundred dollars to say kaddish after her non-frum friend’s father passed away and her friend was immensely grateful. Please, don’t ever undervalue the importance of kaddish.