The Rashba brings proof that Rebbi Akiva himself went against this principle, since he composed Tefillos meant to be said at specific times.
Sam, I listened and I have two Ha’aros. He didn’t mention Achizas Einayim being a type of Kishuf — which is actually Mashma in the Rambam who is Soser himself otherwise. Also, he mentioned Siman together with Lo Se’onenu, but they are separate entities. The Gemara says that although there is no Nichush there is Siman.
However, the point is well illustrated from Rav that tested his Mazal by the ferry. The Gemara does not equate this with saying, ‘a deer ran by so I must return home’. The difference is that one makes sense and is part of an understanding that a day has its Mazal, while the other is baseless.
The Gemara says that we are not supposed to turn to astrologers but that if we heard something from them we should be wary. Being wary on their word is not a violation of Tamim Tihye, since it is true and you are not turning away from Hashem when He actually did make the world this way, with this system. Although, even if true, we are not supposed to turn to the astrologers and it would be a violation of Tamim Tihya.