Reb e – rav boruch ber quotes them in his famous college teshuva in birkas Shmuel, end of kiddushin.
The rashba speaks about making chochmos into a “cowife” of torah, that the torah is jealous.
Knowing math isn’t an affront to Torah, nor is learning it casually. The issue is kevius, which means a set study. A seder. Something you do routinely. Akrai means “here and there”.
The ones who argue with rav boruch bers conclusion (that college is assur) hold that parnosa is a heter, and that they studies are, even though they’re kevius, merely a tool for that parnosa.