I don’t much doubt that there have been attempts to declare it assur since there are all sorts of crazy chumros out there.
The Chayei Adam was not a crazy machmir, c”v. The reason to asser potatoes is because they are used to make baked goods, much like kitniyus were, and this is one the reasons offered for the minhag not to eat kitniyus.
Obviously, most people eat potatoes on Pesach; this Chayei Adam was not accepted.