Beis Din protects the accidental killer from the time they declare he is subject to a revenge killing by the גואל הדם until the time Beis Din successfully gets him to an Ir Miklat.
Also, even when the גואל הדם has the right to kill the the killer, and finds an opportunity to do so (outside the Arei Miklat), the killer has the right to defend himself by killing the גואל הדם in order to prevent the גואל הדם from his immediate attempt at killing him.