Most geirim are looked at as weird and outsiders
Because most of them ARE weird.
And that is the real answer to why they are chozer l’suram. Most of them are not stable.
If this is true, then I have a different possibility. For many years now, I’ve noticed a very big trend in baalei teshuva where after a year or so they begin to backslide. I’ve even seen individuals backslide multiple times.
A number of years ago I mentioned this to one of my rebbeim and I suggested that the problem is that although there is a tremendous effort being put into making baalei teshuvah, there is very little being done to integrate them into their new respective societies. The result is that you can usually spot a baal teshuva from a mile away.
If you are saying that there is a similiar problem with geirim, then maybe the problem is not in the gierus, but how the geirim are taught to integrate afterwards.