I agree that if we were to compare the nisyonos themselves (tznius vs learning/davening) it might be a toss up. A possible proof that men “have it harder” is that chazal say “gadol hahavtacha”, the guarantee (of acquiring olam haba) that Hashem gave to women is greater than that which He gave to men, since men have to learn diligently to earn olam haba, while women just have to enable their sons and husbands to learn.
I think the communal thought process of being more willing to accept the shortcomings of men vs women is due to the aforementioned spiritual and emotional differences between the genders.