Nature is nature. By being around the other person you start to let the guard down and possibly have feelings, even if you know that it should not be.
In a past job, I was put with a team which included a lady who was absolutely Nothing to look at. After working with her for a week or two, she became normal looking to me. I occasionally remember that to remind myself of how we are so impressionable.