“Not that yichus doesn’t have an impact, but there are plenty of cases where the son isn’t worthy and therefore doesn’t take over”
So what? It goes to a nephew or grandson who is more qualified, not to some random, best-qualified candidate with zero family connection.
The point is, it isn’t decided by looking at the entire population and picking out who the most learned and qualified is. It’s decided by looking at the portion of the population with the right yichis (a much smaller number of people) and then seeing who’s most qualified. I’m not saying that’s a bad thing, but it’s kind of silly to pretend it’s not true.