Dear Reb Eliezer,
The first Rambam you quoted speaks strongly against those who decide to immerse himself in Torah and not to work. The second Rambam is that anyone who removes himself from the mundane sphere to serve and worship Hashem, is kodesh kadashim.
The first passage may seem like studying instead of working is idleness. But the second passage is clear that only serving Hashem is less idleness than working. As in, he doesn’t even tend his own fields due to his extreme involvement in Hashem’s service. You picked up from the second Rambam that learning is to avoid idleness. When it is actually saying that it drops all mundane activities, so as not to interfere with Divine Worship.
I have no idea why you were so attacked for merely quoting a Rambam. But the Rambam was himself attacked for it, so there is a mesorah on this topic. I don’t see how one could think that such a strong worded Rambam can be walked back. The discussion should be, whom is the object of the Rambam’s ire?