The Rashbah is used often to rationalize abiding by something without a known source; the Minchas Yitzchak uses in regards to not eating the first piece of a loaf of bread (many do this only on shabbos). Dayan Fisher, though, used to make fun of the minhag, as it was sourceless. He davka ate the first piece.
But we do find sources going back three hundred years about it, though none bring a reason. Kol Davar Shachasu Bah Zekenos – it’s actually a Yerushalmi; I assume that’s where the Rashbah got it from. But where is the line? I don’t know. But I know there is one. And I know that hysteria also isn’t warranted.