I located the source. It’s in the Shulchan Aruch, Siman 210, S’eif 1.
The Mechaber says that if you eat less than a k’zayis of food, you do not make a bracha achrona.
He then goes on to say that there is a safeik if one should say a bracha achrona on a “briyah” (complete fruit), and that therefore, one should avoid eating a “briyah” that is less than a k’zayis.
The Rema points out that if you don’t eat the seed, then it’s not a “briyah”.
The Mishna Brurah points out that if you cut off a bit from the fruit, it’s also not a “briyah”.