“catch yourself”, that is not a mainstream opinion. In fact it is not an opinion within yiddishkeit at all. It is pure apikorsus.
The Rambam, for instance, whom you quote, says the exact opposite of what you claim. He criticizes two groups: The fools who think every medrash must be taken literally, when it’s obvious that some are metaphors or parables, or speak in coded language; and the heretics who dismiss medroshim and say that some of them are false. It seems that you are defending the second group.
Arguments used in vikuchim cannot be relied on — they are not Torah, and often the debaters deliberately used false arguments if they thought they would help them. The object was to win, not to tell the truth. We have examples in the gemara too, where an amora said something to an attacker and the gemara says that he didn’t mean it, he just pushed him off with a straw man because he didn’t want to get into it, or he didn’t want to tell him the truth. So nothing in the vikuchim can be taken for granted.
Pretty much every meforash on chumash will say that certain midrashim are not pshat. Even without the meforshim it’s obvious that that must be true. Midrashim by definition are drush, not pshat; sometimes they help us understand pshat and sometimes they’re irrelevant to the pshat and only address other levels of understanding. For instance it would be strange to try to learn pshat in a posuk from the Zohar. When the Zohar says a pasuk is talking about this or that sefirah, that is true. That is what the posuk is talking about — in addition to everything else. But it’s not the pshuto shel mikra. That’s all the Ibn Ezra and the others you cite mean. But chas veshalom to dismiss any medrash, to say that it isn’t true or relevant.