NO ONE has ever claimed that the Karites are not Jewish. That is not an issue (though many Karites are accused of having convinced the the pre-1917 Russians, and then the Nazis, that they weren’t descended from the ancient Jews and therefore shouldn’t be persecuted). However they have always been considered Jews by mainstream Jews, albeit apikoresim.
If the gittim are not proper, they might be considered mamzerim (assuming their marriages were legal) – some have suggested they be considered goyim since it solves the issue of being mazerim.