RuffRuff
You cannot apply the Aggadaic rule to the Torah.
There are two rules in question: 1) That Aggados are not always meant to be taken literally; and 2) That one may disregard something found in an Aggadic passage if it does not accord with how he sees the world. The second rule, strange as it may sound to you, is explicit in all the sources Charlie and I mentioned.
It is only the latter rule that we do not apply to the Torah, because we believe that Torah is min hashamayim, so obviously one cannot disregard what is contained therein. However, the first rule does apply even to pesukim, at least according to the Rambam who in many cases diverts from the accepted pshat because in his mind it is not ‘rational’ (Probably most notable of this is his position that the whole parsha of the malachim and Avraham Avinu was just a dream), and many other “Pashtanim”, such as Ibn Ezra, who will openly and explicitly argue on a gemara’s explanation of a pasuk because he doesn’t believe it makes sense.
When it is an obvious exaggeration, as in towers in the sky, we don’t take it literally.
Many disagree. And a rationalist could simply respond that to anyone who knows science well, many things are obviously not meant to be taken literally, even though a simple minded person doesn’t realize the strangeness of the statements. This the Rambam says openly in his introduction to the Guide, about Bereishis, and he writes that is was written the way it was, so that simple minded people would have a nice story to read, but someone well-versed in science and philosophy will and should realize that nothing in Maaseh Bereishis is to be taken literally. The Rambam says this in many places.