“That may have been the first time they stopped working for a day en masse, but they certainly didn’t keep the halachos of Shabbos en masse. Shabbos, as you are well aware, is not merely the ceasing of labor. There’s no reason to expect that absent a specific command to do so (which did not exist) that no one wrote a single word, or engaged in borer or even cooked a meal.”
If the midrash sez they kept shabbas, then tehy kept shabbas. Who do you think you are to argue against a midrash?