Klughaussen! If someone erected an idol and then does Teshuva without taking it down, is it worth anything? If you don’t have your friends forgiveness you didn’t rectify the wrong you commited.
The thing here is that you seperate the simple understanding of why these things are important and the deeper understanding of why it is explicit in the Torah. The idea is that certain things are much deeper than meet the eye, that they are everlasting Aveiros. Other Bein Adam Lachaveiro that is not specified in the Torah is important in the general sense of being a Baal Middos Tovos and Ve’ahavta Larei’acha.