Because it’s bad to believe negative things about people that aren’t true. Especially if you disagree with them. Especially if your disagreements are legitimate.
This statement:”the torah is too hard for me therefore i have no obligation or Command to follow it” does not reflect the doctrine of a single Conservative person I’m aware of.