Kaf hachaim says that one shouldn’t even walk towards the door until after aleinu because it looks like the davening is burden to the person. I think the same is said even if one is already outside the shul as long as he doesn’t have in mind to return.
Menu