The simple reading of the Mishna in Avos is that all people are chaviv on account of having been created b’tzelem elokim. I believe there are opinions that it is only talking about yidden, however, it seems the normative position based on the simple reading is that it is talking about all people.
Now, tzelem elokim is understood by the Baalei mussar to mean the koach ha’bechira.
To me that means that the ability to choose right and wrong is what makes us special, because it is that ability which makes us able to lift ourselves up and makes our actions matter.