If you have bad mazel, you might open a window and a crumb of chametz will fly in and land in your food.
That’s why I don’t want to rely on your mazel and I don’t want to eat by you on Pesach.
If you have bad mazel, then maybe the matzos that you baked are for some reason not 100% kosher. That’s why I don’t want to rely on your matzos and I will bake my own, and rely on my own mazel.
(I am just saying this as an example to explain those who are machmir not to mish; I personally am not machmir at all.)