Yichusdik, a good initial approach to funny looking Agadetta is to understand that we don’t undertand was is meant. The next level is to not ignore the Torah Chazal are trying to teach us, but to learn it. Many Sefarim explain Maamarei Chazal on different levels, but ignoring it is not an option.
Your suggestion, not to take it literaly, is exactly what I just did. I explained that it wasn’t our Yom Tov of Pesach and it wasn’t the time of Pesach.
As to the Matza, which is what Chazal say was the intention of the Ugos, Toi answered appropriately enough. The Mitzvos are deeper than merely being a memory. The Maharal writed in Gevuros Hashem, that if it were for the memory alone, there were many options of what to set. Also, he writes, there is no physical memory for the Mann, which is so important that Moshe Rabbeinu had to put some away for the generations. The Maharal writes that these Mitzvos, like Sukka and Matza, are like other Mitzvos — whose reasoning is far beyong us — but it was connected to the memory.
In a straightforward approach, Avraham Avinu had Matza just like we had it in Mitzraim. In Mitzrayim it was for the Emuna of what Hashem will do for us. Avraham Avinu was always praying for the future and his Mitzvos revolved around the future rather than the past.