If it brings one to an improper frame of mind.
Perhaps also on a supernatural level. Rav Moshe says about a tune which was originally used for avodah zarah that although not technically assur to listen to (with no words, in a secular context), it is nevertheless “m’chuar” – disgusting. That might also be true for music originally set to inappropriate lyrics.
True for both non-Jewish and Jewish (such as the now infamous “Zochreini Na”, or “Yidden”, and others) music. What you claim is that an attempt to remove Pritzus (which is a good thing) has extended to include everything under the sun, including the ridiculous (such as Vivaldi). Possibly good for the Hamon Am who doesn’t know better and will listen to dictates from above, but not foe a thinking Yeraim Jew.