Great question. So happy you asked.
A man touching a woman accidentally is not a problem, but b’derech chiba is a problem. Intimate and intentional physical contact between the genders is problematic because it can lead to one of the three aveiros chamuros, namely giluy arayos. That is why it is considered to be a part of them: it’s like a subcategory.
I hope that answered your question.
Besuros tovos! 🙂