When we were in yeshiva years ago, the higher class Rabbeim, teachers, did not speak English very well, and their Torah would be less understood if given in broken English. However, this is how Yiddish was learned in the classroom. Since Yiddish incorporates a lot of Hebrew (and now English) and when in a learning or drasha setting, half of the content is the same as when in English, i.e., quoting chazal; t’nach; talmud; commentaries; etc.) So many English speaking people may lose almost as much even when in English.
But the key is to give your self the opportunity and be eager to learn. Like any language, the more you do it, the easier it becomes and before long, you’re one of them.