I would say that perhaps the distinction is (maybe I’m not sure please don’t shout at me) that it’s actually muttar but there’s a cherem on you that people aren’t allowed to have anything to do with you
but perhaps it’s not an ISSUR derabonon as such and in shomayim you won’t be punished
(not sure at all)