I personally don’t make a big thing out of this issue. Some days I’ll wear a seruga, some days velvet, some days- whatever that other material is.
I do NOT agree with a previous comment suggesting that a kipa seruga means you are less frum. There are kipa seruga people who are much MORE sincere in their frumkeit, AND much bigger learners, than some who will only wear black velvet.
But, nevertheless… some people DO feel that it identifies you with a certain “group” – yeshivish, Zionistic, chassidish, “modern orthodox”, or whatever, and to SOME degree this is true.
Of course there are also some groups that mess up some of this stereotyping.
L’mashal… Many chassidim (at least in Eretz Yisrael) wear a type of white kippa seruga and they are clearly NOT part of the modern OR Zionistic chevra.
ALSO, in Eretz Yisrael, mostly in the so-called “settlements” of Yehuda and Shomron, they mostly wear (large) kipot serugot and they are clearly NOT the “modern orthodox” or “conservatives” that are generally associated with wearing the (smaller) kipot serugot.
Bottom line is there is nothing inherently wrong with a non-black yarmulka or a non-white shirt, but you do have to realize that there are associations that go along with it, so you have to decide if you DO want to make that association, or if you care if others do.