If it brings one to an improper frame of mind.
Perhaps also on a supernatural level. Rav Moshe says about a tune which was originally used for avodah zarah that although not technically assur to listen to (with no words, in a secular context), it is nevertheless “m’chuar” – disgusting. That might also be true for music originally set to inappropriate lyrics.