Something just occurred to me. As I wrote before, I am unsure of the halachos of being mekabel L”H and I have to look into that.
But let’s say you weren’t mekabel the LH. It sounds like you should have considered the guy to have been speaking LH anyhow even without being mekabel the LH. The fact is that the LH had been spread and people were mekabel it. In light of that, the people listening to him speak were hearing his words in that context, rendering the words loshon hora.
As you wrote, everyone considered the Talmid Chacham to have been degredated by his words. Even if the whole story was completely untrue, l’maaseh since it was spread and accepted, the Talmid Chacham was in fact degradated by his words, so his words remain LH, intentional or not.
At least that is how it sounds to me. I may not be understanding the case correctly since it is a bit hard to understand without the details. Does what I am saying make sense to you?