Found another source questioning the literal accuracy of Seder Olam Rabbah:
Rashi to Daniel 11:2.
The Pasuk refers to Daryavesh as Revii after mentioning that three more kings will rise after Daryavesh Hamadai. Rashi quotes Seder Olam that he is calles Revii when you include Daryavesh Hamadai. Then he quotes an alternative interpretation, from the Yosiphun, that there was another king, a son of Koresh, before Achashveirosh.
Now, you realize or course that this is not the first place where Rashi offers an alternative, more simple, interpretation to a Pasuk over a Medrash. As Rashi explains in many places, there are many ways to explain a Pasuk.
You are also obviously aware that never does Rashi refer to this son of Koresh as taking part in the count of years. Here, where Daryavesh is refered to as the fourth, and it would have to be explained, Rashi offers the explanation that the Pasuk could be including this son which is otherwise left out and not counted.
This has absolutely, and unsurpisingly, nothing to do with not taking Seder Olam literally. In fact, as I mentioned earlier, it doesn’t make sense to say that. Seder Olam is not some Medrash relating an experience where we say it must be talking on some other plane. Its purpose is to calculate the years, and that is exactly what the Gemara quotes it for. So, for another reference of where Chazal took Seder Olam literally, see Shabbos 88.