Like many questions, this depends on who you ask. I know frum people who have come on trips with no issues. And there are clearly some who have reasons for concern.
Speak to your Rav, not strangers in the Coffee Room. Regarding Har Herzl, I must object to the comments above. I understand that some have a different hashkafa, and respect your right to follow your rabbanim. Visit there and see for yourself. Speak to the talmidim or Rav Shlomo Zalman. This is a very special place. A place for those who gave their lives for other Jews. Please do not be mevazeh something so holy over political concerns.