It’s time,
That din only comes to play when there is an prohibition of yichud to begin with.
For example, one woman can be in an office together with two kosher men. If one of them are a Parutz then it’s yichud.
Obviously, if the brother or sister is known to have a weakness for this sort of behavior, it would be improper for them to be alone together.