The quote from the Dover Shalom indicates that others aren’t inherently assur.
Also WADR, is the “Dover Shalom” (I believe stories from the Belzer Rebbe) authoritative? I happen to like Belz personally, but that certainly is not the same as a Teshuvah from Rav Moshe.