R Ovadia Yabi’a Omer 6 YD 14 answers exactly this question and says if this is a binary choice, trousers are better
Others add a couple of considerations similar to ones Avira is mentioning:
1) trousers might be OK in some community, but should not be worn in those places where they are not acceptable
2) skirts might have become a symbol analogous to the kippah – this is how Jewish ladies identify themselves as observant. My projection from the use of kippah analogy: you could and should use it generally in friendly atmosphere, but OK to avoid when you are in non-friendly environment