Ubiq – great question; the meforshim discuss it at length.
The simple answer is that there’s a difference between situations which lead to issur in and of themselves (i.e. yichud, or in our case, shmiras aynayim) and activities which are mitzvos (mesamayach choson vekalah)
A deeper answer is found in likutei moharan, which discusses the different types of yatzer horas of different types of tzadikim