“there is never a case where I can a priori know that not vaccinating is the right option or even close to a legitimate option. “
Actually there are such rare cases, mostly allergic reactions to a previous dose of the same vaccine. The number of such cases is vanishingly small. But for everyone else not covered by a specific contraindication, it is always better to vaccinate as there are no longer any vaccines given in the US that have major risks. (Indeed the form of the polio and pertussis vaccines have been changed to make them less effective in order to reduce the already low risk of severe adverse side effects down to essentially zero.)