Gadol, it’s that exact notion of “maturity”, “strength”, “conviction”, etc.., which halacha does not permit. Many people who are in that mindset believe that yichud isn’t necessary for them, because “do you think I’m going to….” – curiosity/engaging with bad ideas (why the quotation? Are anti torah ideas not bad?)
Halacha understands the natural weaknesses of people – even, and especially great people – “anyone who’s bigger than their fellow, their yatzer hora is greater”, and he needs more protection, not less. The steipler davened for rav chaim all his life that he not fall into foreign influences, well into his gadol-hood(orchos rabbeinu, i can find the page if necessary). It’s false bravado and arrogance to think one is above sin – regarding apikorsus, chazal say “shaani minus, de’mashchi”, heresy is different, because it drags (one into it).
Regarding lectures with forbidden content, some say that one can tune out those parts and be “masiacb daas”, but with the availability of kosher, curated college courses, it is difficult to justify it even according to those opinions. The majority, however, hold that such study is assur. The fact that “the velt” did it in the 60s/70s, etc, on its own, doesn’t permit it. People had many nisyonos in those days, and contrary to your point of the “thousands of frum people who remained frum” a very, very large number did not. More people went off the derech at that time than now, and college was not a small factor in that. Besides going off, many were/are weakened by it and become much less observant. Look at Mike Moscowits – he was a frum , orthodox rabbi who went to college campuses to do kiruv. He was dragged into the gender bender world and now, with his hat and jacket, promotes LGBT agenda and works with reform/conservative groups in that capacity.