It just can’t be an independent Tzaar (like thinking about dragging blankets while eating):
But if I eat in my house, it’s not in my face. So you should pattur that “mitzaer” too.
Sun or cold doesn’t come from the Sukkah itself
Semantics. The sukkah is hot or cold or wet, the house is comfortable. So the discomfort does come from being in the sukkah.
Also, did you address my second point about how this “heter” even l’shitoscha doesn’t apply to at least the vast majority of chasidei Chabad who don’t sleep in the sukkah?