There’s a lot that goes into shidduchim besides dress size. Looks/appearance is based on overall presentation, which includes dress (not size, but whether style is complimentary), communication style(whiny complaining or shrill voices don’t cut it), confidence (posture/how a person holds themselves)…
A girl who’s average weight, not super thin, but dresses very well, is warm engaging and outgoing, and stands with great posture (took ballet, is graceful), will turn more heads and invite more opportunities.
A girl who isn’t obese, but her hair has no style, she dresses like a much older woman will rarely invite offers for dates.
There’s nothing wrong with turning down a shidduch because of lack of attraction. Personally, I think it’s ridiculously shallow to decide to only be attracted to ultra skinny girls. It’s even sadder for the future wives of these men who then have to spend their lives trying to live up to an image. I don’t buy that it’s how men are built (only attracted to size 2’s). I think it’s an indulgence that some men allow themselves. Have a talk with Rabbi Goldwasser who counsels women with eating disorders, and see how fast he tells the guys that that belief is nurishkeit.