Bad idea. Plain and simply said, it is wrong and illegal. Straight out of the Voting Rights Act of 1965:
(1) Congress hereby declares that to secure the rights under the fourteenth amendment of persons educated in American-flag schools in which the predominant classroom language was other than English, it is necessary to prohibit the States from conditioning the right to vote of such persons on ability to read, write, understand, or interpret any matter in the English language.
(2) No person who demonstrates that he has successfully completed the sixth primary grade in a public school in, or a private school accredited by, any State or territory, the District of Columbia, or the Commonwealth of Puerto Rico in which the predominant classroom language was other than English, shall be denied the right to vote in any Federal, State, or local election because of his inability to read, write, understand, or interpret any matter in the English language, except that, in States in which State law provides that a different level of education is presumptive of literacy, he shall demonstrate that he has successfully completed an equivalent level of education in a public school in, or a private school accredited by, any State or territory, the District of Columbia, or the Commonwealth of Puerto Rico in which the predominant classroom language was other than English.
Besides, I would certainly see this working against every single person who speaks Yiddish or Yeshivish as their primary language.