Nothing is glatt in this situation. SOME 20 year old boys have the wisdom and maturity to be able to retain an open door with such a friend, and even potentially influence them positively. Others ABSOLUTELY cannot and should not remain in such a friendship, because they themselves are easily influenced. You have to know your child and his kochos.
Sara Emeinu saw Yishmael acting in a way that was detrimental to Yitzchak and took immediate steps to distance them from each other. Avraham Avinu didn’t want to send Yishmael away (i.e. if you break off your son’s friendship of many years), but Hashem told him explicitly to listen to all that Sara said.
This boy is not a Yishmael chas v’sholom, and your son is not Yitzchak Avinu. But you are wise to be concerned, while at the same time you should recognize the potential your son could have to remain a stable, positive influence in this boy’s life. I don’t have an easy answer for you, but keep the lines of communication with your own son open.