Most of the statutes were put up by opponents of the civil rights movement in the 1950s and 1960s, as a symbol of resistance to desegregation, blacks’ voting rights, etc. This is a good argument for taking them down. Some of them were put up for a reconciliation-oriented purpose, including I think the one in Charlottesville. This would suggest being selective in which ones we take down.
In any case, having them prominently displayed gives them honor and respect which they do not deserve. Taking them down and putting them in museums instead, as historical items instead of public monuments, is a popular proposal on the right and left. There are decent arguments on both sides.