Who wants to get rid of him (as opposed to annoy him)?
If Trump leaves office, Pence becomes president. He is a traditional conservative Republican with good ties to all the non-Trump parts of the Republican party. In fact he was mentioned by the “never Trump Republicans” as a possible “savior” in 2016, and was chosen as Vice-President to unify the party. If Trump were forced from office, his supporters (the “deplorables”) would be outraged and would transfer their support to Trump’s loyal Vice-President, meaning the Republicans would go into 2020 unified. The last thing the Democrats want is to get rid of Trump.
However if Republicans “ditch” Trump it would alienate the “deplorables”, and the rest of the Republicans aren’t strong enough to win in 2020 without them.
And in all fairness, the worse “crime” Trump is accused of involves the same sort of activities Clinton was guilty of, and which are clearly not “high crimes” as defined by the constitution.