The gentle approach is always the best. With a smile just gently say that you are having a problem concentrating or explain that as you have so many distractions during the week that when you get to shule on Shabbos it is really your only restful time to communicate with Hashem.
Whomever you speak to already knows about assur lidaber bishas hatefilla. I have been to shules and people scream SHaa, NU or quiet or give loud claps. They make just as much a disturbance as the people talking, come off as meddlesome even though they are in the right, and nobody really listens. Eventually calm demeanor wins over the respect of others and when people see you, people will know that you are someone that does not like talking during Davening. They will automatically clam up when they see you and perhaps change their own behavior from your example.
B) I ask you to consider your opening paragraphs in describing the situation you faced in Shule.
” davened in a certain shule this past shabbos.” Does that mean you were a visitor? You also wrote “The scary part is that the Rav ….?”
It seems to me that you were a guest.If so it is basically impossible to judge a Rav or the people that Daven there. Even if you are a regular perhaps the Rav is working behind the scenes.
Don’t get me wrong whether a guest or a regular you are entitled to a normal Davening, but perhaps the Rabbi is working on this Mida with his shule and you dont know about it? And perhaps even some of the perpetrators themselves are trying very hard not to talk.
I have been a guest by family, friends, neighbors… on Shabbos.Sometimes the same kids act out repeatedly in the same manner,and you wonder why the parents are not disciplining differently.(similar to the Rabbi this is a main function of their job) but of course we don’t fully know what is going on behind the scenes based on our short visit.As an outsider looking in it is not always possible to get a clear picture.
In some extreme examples I know of two people with short term memory issues, one with a mild social anxiety disorder,and a few people with major tzaros that just want to blow off steam to their friends. (these are people whom if you saw them on the surface in a million years you would never suspect such things are going on)These people truly struggle with keeping quiet in shule and it is a real nisayon. It is very hard to know how others are truly working hard on their midos.
May Hashem see that we are truly Amech Kulam Tzaddikim in what other nation in this day and age would you have people complaining and working on a Mida about talking in the presence of a G-d that can’t be tangibly seen or heard?
Tizku Limtzvos!