I once kept having trouble getting certain thoughts about something I had used to do out of my head. They would pop up at the most random times. I asked one of my rebbeim about it and he told me it was because I still felt an attachment to it.
Time has proven him right. At the time I was still shaky in my decision to not do that particular thing, but once my decision solidified and the nisayon passed, I moved on and everything related to it left my mind. I can still actively recall those things if I want, but its not there hovering in the back of my mind all the time.
My advice is to commit to your present path and let go of the past. Eventually, you will lose your connection to it and become a different person. The person you want to be.