Yehudayona but it can’t be all there is to it, for example string beans are kitniyot forbidden to some, even though they are obviously not usable to prepare bread-like foods, and they are forbidden even if we have them picked before any seed is visible in the pod.
Also, could you think of bread made of coriander flour? Cumin flour? Differently from corn they were known and permitted, yet some have the minhag to avoid them.