Eclipse,
You are not alone.
I’ve been in somewhat similar situations and yes, it’s very painful indeed. That’s why many people are not educated enough on how serious this transgression is. (I was ironically reviewing this topic in “Sha’arei Teshuvah” this morning.)
Under no circumstances am I saying that this is your case, perhaps but I personally I used to get extremely paranoid sometimes about what others think of me from experiences I’ve had in the past either with them or people they know. It came to the point where I moved out of the community. I will say that after a few years, I sucked my pride in and went back to the same yeshivah and community with the mindset that I wasn’t and will not let anyone make me feel terrible for being who I am and that I have every right, just as they, to be there as a part of the community. They can say whatever they want. It will be on their cheshban at the end of the day.
Once again, I am sorry to hear your situation. Keep in mind that you do what you need to do and God is running the show. There are reasons why things happen that are out of our control but IY”H we deal with it the best we can.
Best,
Yirmiyahu