All the sifrei kabballah that talk about olamos are talking about spiritual worlds not physical ones. That is clear from how they describe some of these worlds.
If you look in Rashi on that posuk in shoftim it will be clear that the “inhabitants” are those that are in the vicinity of the star.
A chariot of fire, is a chariot of fire. You need to bring some type of proof that the words mean something they doesn’t say.
The entire world was created for the sake of klal Yisroel and the Torah. We are here and the Torah is with us. Without a mekor that specifically speaks about a physical world populated with physical beings there is no reason to assume that one exists.