Well, to start, the Ramban says so explicitly in Seder HaVikuach, and so does Rav Shmuel HaNagid in Mevo HaTalmud. (It’s worth noting that, although not a Rishon, Rav Hirsch followed this view as well, as he writes in Collected Writings)
The Ibn Ezra disagrees with Midrashim countless times in his Peirush on Chumash, as does the Ramban, as well as the Radak in his Peirush on Nach.
The Ritva says that the whole story of what happened during the debate about Tanur Shel Achnai didn’t really happen.
The Rambam in his commentary to Mishnayos writes that whoever thinks that everything Chazal say must be taken literally is a fool who makes the Chachamim seen like fools.
The list goes on.
I know there are those who argue, and that’s fine. Just don’t pretend this isn’t a mainstream opinion.