If your original agreement was that you do not have to work on Pesach then it is unethical of him to go now and use that as a means of pressuring you into working on Sunday. If he does so he is unfairly taking advantage of you. Of course, he can ask you for whatever favor he wants, but he shouldn’t be guilt-tripping you based on something he has already agreed to.
If per your original agreement you are supposed to work on Sunday, then, well, why would you feel that you don’t have to work? There is still nothing wrong with asking for a favor, of course.