The fact that you’re impressed by ostentatious looking homes proves the point of this moshol- that the outside of the homes are for the people walking by and not for the people living inside.
The people inside hardly get to see the outside. What is giving them “enjoyment”? The fact that the people outside are impressed.
And if this is so- why shouldn’t the outsiders have a say in how it looks? (this is just to bring the point home)