yoyo,
you know first hand the reason why it’s wrong. actually, consider yourself lucky to have stopped when you did. it could’ve gotten a lot worse.
The Yetser Hara in that matter is extremely strong. It will take you places where you should not be, make you do things that you should not do, and make you have feeling that you should not have. One small step can lead you down a slippery slope. You cannot fight it; no one can. That’s why you read about great rabbis that assiduously avoided women. With their learning and piety, they deeply understood the power of the Yetser Hara. They knew that even they might not come out unscathed. Think of a big buy fighting with a small guy. The gadol might win, but still might get bruised.
We are not such gedolim, and we may easily lose such a fight, so we must be even more diligent in our kedusha and prishut.