Sam- Why do you say there is no ain maavirin? Are you aware that the Yerushalmy holds it is a Mitzvah to remove tfillin and they even used to make a Bracha upon removing it? If you use the right hand to reach up to the shel Rosh, you pass the shel yad on the left hand. If so, you should remove that one first. Isn’t that ain maavirin? If however you use the left then you avoid ain maavirin!
Additionally, using the weaker hand demonstrates a reluctance to remove them.