IF he indeed claimed himself to be moshiach, as many (probably most) Chabad chassidim claim, and as posted by several in this and other threads, then there are two options: he’s moshiach, or he was a moshiach sheker.
Where do you find in Torah such a concept as “moshiach sheker”, or the idea that there’s something wrong with it? “Novi sheker” is a genuine halachic category; “moshiach sheker”, as fa as I can tell, is an invention of the maskilim.
Shabsai Tzvi and Yaakov Frenk were rejected, not because they turned out not to be moshiach, but because they became resho’im, openly rejected observance of halacha and told their followers to commit serious aveiros, and eventually committed shmad.
But the only halachic source we have for hilchos moshiach is the Rambam, and he holds that Bar Kochva was a tzadik and should be admired even though he failed as moshiach. Nor have I ever heard that gedolei yisoel ever condemned Shlomo Molcho or Dovid Reuveni or any of the other people who tried to be moshiach and failed, but did not go off the derech.
So even if the Lubavitcher Rebbe had publicly announced (which he didn’t) that he was moshiach, and then he failed, there would be notihing wrong with that.