LMA: Rav Moshe actually mentions both, and seems to take issue only with eating things, not entering (to use Sherutim, for example). And as Rav Moshe seeems to say there (last 5 lines), if everyone who sees you will know that you are doing something unquestionably Muttar, then it would be allowed.
Why don’t you take a look and decide for yourself?