I am expanding on R Soloveichik’s quip that we discussed before: he learned philosophy while other talmidei chachamim (sic!) were saying lashon hara about him.
Take a broader view here: learning chol is an aveira by some and in some circumstances and a worthy thing to do by others, depending on circumstances. There are a lot of bitul zman options that are negative without having a possible positive side. So, start by eliminating them.