Nechoma,
Maybe the reason that people are susceptible to improper thoughts is because we hyper sexualize everything. If you say that a skirt much be 4 inches below the knee, then if a man sees a woman in a skirt 3 inches below the knee, he will have improper thoughts. If sitting in the same woman is considered taboo(non yichud situation), then when are in a room with a woman, you will have those thoughts.
We obviously have to take this in the context of halacha, but we must realize that unnecessary chumros might be hurting us more than they are helping us.