“Teachers in frum schools aren’t often trained to handle challenging kids. They usually make situations that are dealable worse.”
That is why all teachers (and Rabbonim) should be qualified teachers. Even if the school has no money for special ed teachers, a qualified teacher is trained to pick up these problems and together with the parents, possibly work out a solution.
Also, much as it will pain some parents, occasionally there is a child who is not cut out for a yeshivah or any day school. Use the same tuition money to get the best quality education that you can for this child – even they must learn their yidishkeit at home. As bymeidel said even all children in one family cannot do the same work. If one child is unable to cope with yeshivah life, then send them elsewhere.