I think it depends on what caused you to change your yarmulka. Some people don’t care whether they’re wearing velvet or those cardboard ones they give out at the kosel. To them they aren’t going up or down.
Children that come from Chareidi families that fall to the side C”V may change their yarmulkas as an expression of their falling. Whether you believe there is a difference in yarmulkes or not, in such a case, the kavanah of the child changes the status of the yarmulka.