Conversion and Trespass to chattels. They’re terms of art for various level of damaging other people’s property. They probably think it’s their land, so prepare for some ordinary trespass. If they really hate you, they might describe it as misdemeanor theft and call the cops. Could be some vandalism charges. Just because its on your land doesn’t mean you can break it. (Of course, in jurisdictions allow self-help, you can do that. As I’ve said before, check.)
In all likelihood, your damages for trespass are nominal, meaning you’ll recover $1 and an injunction compelling him to stop. You might punitive if they find he acted in bad faith. If you damage the boards on the fence, you will probably cause more than $1 in damages. If, as a consequence of a court might see as your illegal destruction of his wall, he suffers further damages (someone breaks into his work site and steals his tools) you might be on the hook for that. Etc. And it looks bad at trial.
Why give the other side ammunition when you give them a preliminary injunction?