In English, “Schwartze” is definitely a perjorative. In English, the language of assimilated Jews, “black” is bad. Indeed, the reason words such as “Negro” (which is a latin root) or “colored” were used to refer to African Americans was that “black” was perjorative since it meant “evil.”
In Yiddish, or “yeshivish” it isn’t clear. Yiddish has far fewer words than English, and there are no ways to say “Black” other than “Schvartz”. Any of the Americans eupemisms sound alien and foreign sounding (and some secular Yiddish texts tried to introduce the word “Negro” into Yiddish, without success). Persumably if you are refering to whites as “Veissers” there shouldn’t be problem using “black” in that context.