I went back to work when my first was a toddler, but I think it was a mistake. The babysitter was a very poor disciplinarian and used to beg him in a whiny voice to listen to her. We had our hands full after that. I later chose to be a SAHM.
My sister also had a nightmarish experience with her babysitter after returning to work. She returned home one day to overhear the babysitter insisting that my niece call HER mommy. That arrangement ended very quickly.