Ever heard of the Gemara where the Amora’im said to one another let’s go down this street where the Beis Zonos is, and get Schar?
Yes, Ein Aputrapus La’arayos, but you do know what you are holding by, and what is not a Nessayon. In fact, the Mishna says about certain situations that they are not a problem of Yichud because nobody is expected to do such stuff.