laskern, That is the generally accepted minhag where the shul minhag is NOT to wear tfilin. In a shul where the minhag IS to wear tfilin, e.g. Yekkish or Litvish, generally the non-tfilin wearers daven in the main shul. The reasoning is because often it does happen that people daven without tfilin for various reasons. Therefore it is not obvious that they are differing from the shul’s minhag.
hml, very interesting. But the story needs context. Are you chassidish or sfardi? Are you in EY? etc.