It clearly is annoying. I personally would consider it an opportunity to work on my own Kavanah and skills at concentrating and becoming immersed in my Tefilos enough to not notice outside distractions of any sort, such as this, or talking, or construction outside.
But if this fails certainly the best approach would be to tell the Rav of the Shul of your problem and ask him to help if it is possible.
I wouldn’t say anything to the disturber myself. Unless you are better at speaking to people than me, most likely nothing will be accomplished except frustration, further anger and animosity between two Yidden, whether you are right, or not.