Most geirim are looked at as weird and outsiders
Because most of them ARE weird.
And that is the real answer to why they are chozer l’suram. Most of them are not stable.
And it gives a bad name to the minority who are normal and stable, when we don’t differentiate between them. The rabbonim should be able to tell the difference (I can), and if they cannot, they should get out of the business.