you must be a man,cause most ladies dont mind being invited and no need for cooking and prepare for Yom Tov.
1. Why is it so bad to go?
2. If you dont want, why do you feel an obligation?
3. Would you be happy to invite them, or you want a quite Yom Tov?
Just to let you know, many friends who are married to a large family, and in laws are tired of preparing a hugh YomTov would love to be invited, but dont have the option. If you feel you want some quite time with wife and kids, I totally udnerstand. You can! would be nice if you help wife with preparation more than you usually do, since its a lot of additioanl work and stress for her to do it at home. Have a great Yom Tov!