Counting numbers is not infinite because no one can get to infinity through counting numbers. In fact, no one can get to infinity from any finite point. That is what the Chovos Halevavos was getting at.
Using infinity in math is a toy, since there is no relationship between it and the finite, while there is no mathematical reason for such a concept not to exist. This is why we will find paradoxes in many levels.
This doesn’t mean we can’t understand what is meant by infinity. It simply means that there is no bridge to get there, and any attempt to define the infinite by applying finite terms will have paradoxes.