Wisey,
I believe that the answer to your question is that the mathematical “infinity” doesn’t actually exist. The number line is a human construct, it isn’t a real thing. Actual infinity is impossible for the human mind to comprehend.
Example: Imagine an infinite number of marbles. Now remove one marble. How many marbles are left? The answer has to be less than infinity, which will therefore be a number. Now put the marble back. You won’t have infinity, you will have the previous number +1.
In other words when dealing with actual infinity, if you break it down in component parts then you will quickly run into the above paradox.