There are very very few people who have such difficulty with masks that they cannot wear them. Those people should stay away from others.
I think the proof that it’s more common than you claim is that so many people are resistant.
Why should those who can’t wear masks stay away from other people? Maybe the more vulnerable people should stay away from others.