write or wrong I do deeply empathise, I understand the importance for you of fitting in with your community. I agree with zahavsdad about the tragedy of lashon horah and imposed conformity that afflicts many a tight knit observant community.
Just a word of encouragement from the heart, whatever you feel inside do not give negative people the chance to get busy with your family set up, be confident and hold your head up in public, negative types can sense when we ache for our children.
Ok you are in a difficult challenging place now, well it is not easy to be a parent, the trials and tribulations of those around you may be far worse but better hidden. I never realised this until I ended up teaching young adults!