What if you are davening at the earliest possible minyan (i.e. at sunrise)? What you are saying would not apply. The fact is, that the halacha for a person who is yoitze laderech is different than for a batlan.
In the winter, sunrise is often not early enough for people who work a full day. I know of a minyan that davens before sunrise during the winter specifically to meet that need (as opposed to many shuls that constantly change the time of the first minyan so that it is never before hanetz hachama). Normative halacha permits a yoitze ladreceh to daven before sunrise b’yechidus- k”v with a minyan.