even if it was mutar to ascend har habayis, there are many other sins that are done when a person ascends other then the one main sin of not ascending har habayis.
sins include from
1)not listening to your Leaders-written straight in the torah
2)going to a place of avoda zara to others (as a person may not go to the synagogue where yoshke was killed although it was a synagogue & not a church etc…)
3)many other sins involved
what would you answer to Hashem regarding these other sins if YOU ascended har habayis C”V