Originally “Colored” was the polite way to say “Black”. Today Black politicians use the phrase in order to make their group seem bigger. Various other groups defined as non-white use the phrase to argue they are really oppressed. One problem is that some persons of European ancestry are darker than many “people of color”. The biggest problem is that when you average in American born Hispanics and especially Asians, the average suggests “People of color” have the same economic level as “Whites”, thereby roving that racial discrimination is NOT a major factor in America.
Like all “racial” terms, one should assume the users are fools and racists, and also probable anti-religious (since on a fundamental level they are rejecting the idea that HaShem created the human race and that all people are closely related).