Tznius is also ben adam lechavero. Who knows how many men can see a woman dressed inappropriately and have taavos?! It is definitely something which affects others.
Additionally, tznius is not something which is done in private. What you wear is seen by all and makes a statement and makes your clothes more acceptable.
No- I don’t think men should be looking to notice the difference. But even though it is a challenge, as women, that is our job to overcome it.