As a BT who never even heard of Yiddish, I grew up with my Sephardi parents speaking Ladino to each other so that the children wouldn’t understand what they were talking about. I knew of nothing other than that and English during my childhood years. I was actually very insulted when a friend in my 20s called Yiddish the “Jewish language” as if Ladino was worthless. I think it was just a matter of lack of exposure to the other.
As my life changed, I had no idea that I would end up having all of my children go to Yiddish-speaking schools. When the oldest was about to start, I decided that I’d better do something about learning the language a bit just so that I’d at least be able to understand what my kid was learning. I found a class and went to it for a few months and it really helped. I definitely don’t speak like my kids do, but I was able to understand most of the school play my daughter was in recently and even a shiur given by the principal as well.
I think the rabbeim who speak in Yiddish do so to avoid speaking in loshon kodoesh, and do so instead of speaking in Aramaic, which no one does at all these days. They won’t speak in Ivrit because of its problematic origins.
What’s the problem with people speaking the language that they feel comfortable speaking in? Do you have a problem with those from France speaking in French? Are they allowed to teach it to their children even though they don’t go to French schools? Can I teach my children English even though we live in EY? If you want to find problems, then you’ll be busy pointing fingers. Do what was suggested above and learn a bit of the language and you’ll find that you feel less uncomfortable, it worked for me as well.