LBJ, I think the Vilna Gaon says that touching hafrashas af does not require washing hands, but touching hafrashas af does.
Sam, thanks for correcting my terminology, but I still think there was more to it for my sake than just the concern about the pieces going on the floor or washing my hands, since he could have just told me to be cautious about those two inyanim. While it was many years ago that I spoke to him about this, I still recall the meeting and I felt that it was something that was akin to timtum halev if I ingested them (the pieces are so small, do you really throw them all out afterwards?).