Sefardim came from Spain (hence “Sefardim”) before they moved to the Middle East.
First of all, some Sephardim went from the middle east to Spain and lived there for a much shorter time than Ashkenazim lived in Europe.
2nd, not all “Sephardim” came from Spain.