Then don’t tell him/her that you got a call about him/her. They might start asking you what you said, and it could be very awkward. Even if you think you can avoid letting on that you said something negative, there’s a very good chance that they might figure it out or at least suspect something. (have you ever read “Diamond Dust” by Reva Pomerantz?)
Or they may find out that the guy heard xyz about them and put two and two together and figure out it was you.
It actually happened to me at least once that someone told me that he told the guy negative things about me. And it wasn’t even the reference – it was the shadchan!