Your premise that the children of a Challal are Assur le’Kohen is incorrect. See Shulchan Aruch Even HaEzer 7 16.
Well, he’s only partially incorrect. Any daughters of male-line descendants of challalim are forbidden to Kohanim. So, the OP’s question is still valid.
The Wolf