Ct, YU had a lot of influences before, during and after rabbi yoshe ber that were much more modern then he was. Rav elchanan visited it and torah vodaas, and remarked that YU is a bad Yeshiva with a good name, and that torah vodaas is a good Yeshiva with a bad name (the name implied that there’s something more than torah… It was given that name to make a board member happy who attended such a school in Europe..YU was named for rav yitzchok elchonon spekter).
Rabbi yoshe ber was responsible for only some of the problems in YU, but that isn’t really relevant to the view that gedolim had of him personally. He was told by rav shraga feivel that he could actually be a rosh yeshiva in torah vodaas if his wife would cover her hair, he would stop going to operas, and a 3rs condition that is supposed to be kept secret