That’s a pretty loaded question.
I think as long as it is only a bad influence, I would keep going. The main influence kids get is from their parents- I’m not very concerned about outside influences. (Yes, that is the truth. If outside influences were such an issue, you would see higher rates of kids going off the derech in 5 towns than in williamsburg. Nobody has ever made such a claim.)
However, I am more concerned about him scaring your kids. Your kids do not need to be subjected to abuse, neither to maintain your friendship, nor to encourage the other family who was kind enough to take their brother in.