The stones in the Kosel are Herodian stones. The Kpsel eas not part of the Bais Hamikdosh, just part of the exterioir retaing wall.
People are somewhat spoiled by the current Plaza. The Wall was never treated with respect by the Muslims and was always used as a building wall for other structures. If you see pictures of the Wall pre-1967, it’s rather unimpressive; a small exposure on a narrow street filled with Arab housing. Jews get to the site of the Bais Hamikdash
By the way, if you take the Tunnel tour, the still unexposed portion continues down to where the corner fortress was, and you walk on actual Herodian street that paralled the Wall.