There is no clear rule for these things. It depends on who is person is. If you’re not comfortable, there’s probably a good reason for that. Even if you are, maybe you shouldn’t be.
Who says you have a right to be frum on yenem’s cheshbon?
Suppose you don’t have tefillin and only your neighbor’s wife is home? You have to knock on the door, ask for the tefillin, and put them on. And deal with your nisyonos. You can’t not do mitzvos because you have nisyonos.