Ask him on date 3 (or 4), but ask the one he broke off with before you even date. It might be for a dumb reason (his mother serves Shalesh Seudos on plastic plates), and its the kind of thing that you could give two hoots about.
Then again, he might be Freddy Kruger’s understudy, so you might want to bring along running shoes!