could be a sensitivity, a personal feeling, or even a fear. could be a kabalah or gezaira they set up for themselves because they have a certain weakness. many other similar explanations. and they didnt walk out because you were in it, they walked out because a woman was in it.
do you take it personally when all the birds fly away and the squirrels run when you walk by? theyre running from a person, not from you.