Sam, just do the math. You concede there are things one party can do that are evil enought to justify the other side withholding halachic divorce. So then in each situation, either the one side is being evil or the other side is.
You assume it is very rare that the one party would be so evil–well so should you assume it is very rare that the other side would be so evil. Equally rare.
I assume about half of killings among people who know each other are justified.