“Suppose you don’t have tefillin and only your neighbor’s wife is home? You have to knock on the door, ask for the tefillin, and put them on. And deal with your nisyonos. You can’t not do mitzvos because you have nisyonos.”
If he really feels that it would be a problem for him, it is assur for him to knock on the door, and a bigger Mitzvah to skip putting on Tefillin. In this case, putting on Tefillin would be a “Mitzvah haba al yidei aveira”.