There was a nice story in Smiling Each Day from Rabbi Twersky about a wealthy guy who is bragging to his Rebbe about how he is mistapeik b’muat and only eats hard bread and drinks water. To which the Rebbe replies (something to this effect), “From now on, I want you to eat stuffed geese and drink fine wine. Reason being, if hard bread and water are good enough for you, you’ll think stones are good enough for the pauper. But your mitzvah is to give generously to the poor, and in order to do that, you need to be eating better yourself.”
I tend to agree with old man – someone else’s money should stay out of your mind. Yet at the same time, I think there is something about farginning that needs to be worked on. I would hope frum Yidden could walk past the luxuries owned by the rich and say, “Thank you Hashem for giving my fellow Yidden parnassah.” This is the achdus we should feel – I am happy for my brother that he is successful, and I wish more of my brothers were.