Rabbi Akiva’s wife was willing. If your wife and family needs you, and you leave to learn, then its abandonment.
I know some men who use learning as an excuse to avoid homework/dinner/bath/bedtime. If you are doing that, chances are your learning has very little meaning.
Make sure your family is taken care of. They are your primary responsibility.
Remember, chinuch is not just about making sure your sons can learn gemara. They need to learn how to be a father, husband, and good person. Learning is absolutely a huge part of that, but being there for your family is also important.
You need to find a balance for yourself, your wife and your kids.