bpt, it’s with logic like yours that is partially what is so wrong with the world today. The logic of if i don’t hurt you then i can do whatever i want.
Fact is, the torah says we have to look out for each other. If we’re on a boat and i decided to drill a hole in my room we’re all going to sink.