Some people have a very hard time with language. This only gets worse as you grow older. There are plenty of Shiurim of all types. Find the right flavor of Halacha Shiurim to follow and within a few months you’ll be able to out-maneuver most of your peers. That’ll do away with the Am Haaretz feelings.
But parallel to this it is still worthwhile to make a steady effort in learning real Sefarim. Start with Chumash. If you are comfortable with Chumash it shouldn’t be a big deal to move to Mishnayos with Rav/Tiferes Yisroel.
However, you should desperate these two distinct efforts. Don’t let the slow progress of one get in the way of the other.