Gilgul is from Toras Nistar. It wasn’t mentioned openly until after the Ramban. The Ramban himself spoke about it but refused to do so openly. Therefore, lack of point blank reference in the Gemara is not an argument. However, the Ramban does read this into the words of the Gemara.
Any and all students of Kabbalah know of and hold of Gilgul. It is in the Zohar Hakadosh. Without Kabbalah we wouldn’t know of it, wrote the Ramban. To quote a Gadol who might have been unaware of Kabbalah, and is drawing exclusively from Nigleh, does not prove any point.
Moreover, as I said, it doesn’t solve the whole issue. It is merely one piece in the puzzle. Also, this is used to answer Tzaddik Vera Lo, which is very uncommon, according to the Ramban.
Don’t view it as if this guy did something wrong that he, or anyone, doesn’t remember. He has a Neshama that needs rectification, because of whoever had it last. See again what I quoted from the Ben Ish Chai.
Yissurim Shel Ahava also serves a purpose. Why would you feel bad for the guy. Do you feel bad for a laborer who is getting paid nicely? On the other hand, even if know how someone actually earned a punishment that is not a reason not to feel bad for him. We feel bad that he did whatever it is that he did, and earned and received this punishment. That is how the Medrash Eicha describes the crying of HKBH.
Besides, this person is not a Rasha now no matter what he was in the past life. Should we view him any worse than a person who did Teshuva? Often, they also get Yesurim to be Mechaper or to elevate them, which is part of Yesurim Shel Ahava.