I believe that if it is a nice day it would be fine, but if it’s overly cold, or to sleep, would be problematic, because it seems that you are there because of the mitzva. This is why we pasken to eat but not sleep in the sukka on Shmini Atzeres (in chutz laaretz), because eating is not obviously abnormal but sleeping is.
This reasoning is further extended to explain why many Chasidim do not eat in the Sukka on Shmini Atzeres, or eat only during the day, contrary to the psak of the Gemara. The explanation given is that while in ??? it was warm enough to eat in the Sukka and not arouse suspicion, in Eastern Europe it was frigid and thus being done obviously for the mitzva, which would be problematic because of ?? ?????.