The Brisker Rosh Yeshiva was referring to those who have adopted the mores of bnei Torah and then dropped them.
As for Rav Chaim shlita, a baal habayis renowned for also being a distinguished talmid chochom, when making aliya asked Rav Chaim if he should adopt the custom of always wearing a white shirt and not a blue shirt during the week and a white shirt on Shabbos. Rav Chaim looked at him and and said “it suits you”