Add to Charliehall’s list R’ Hai Gaon and R’ Shreira Gaon in Otzar HaGeonim Chagigah pp. 59-60.
One can find the Rambam’s words on this in Perek Chelek, but he most poignantly expresses his view in Teshuvos HaRambam (Jerusalem: Mekize Nirdamim, 1989) p. 739, where he writes: “We do not pose difficulties in the Aggadah. Are they words of Tradition or expressions of reason? Rather each individual considers their explanation as it seems fit to him. In this [Aggadah]
The Ramban openly expresses this view in his vikuach which can be found in the Chavel Kisvei HaRamban, Vol. 1, p. 308. See R’ Chavel’s note on the bottom.
There are more sources, but it is quite clear that at least in the times of the geonim and the rishonim it was not considered kefirah to believe that aggados are not meant to be taken literally or even to deny things which are found within them.