mentsch1,
Here’s a simple example of how one is supposed to worry more about the gashmius of others than their ruchnius: When inviting guests, one should not say, “It would be wrong of me to serve them a sumptuous meal because that would encourage teivah and physical pleasure”. One can limit the foods he or she eats and thereby minimize their own physical pleasure but one should not do so to others.
A true baal musar works on himself and isn’t so focused on giving tochacha to others, especially in matters where he is weak.