Or maybe, maybe, we should read Chassid Yavetz in full and not just on popular lines.
Here is a different quote from R Frand’s book: one of the Spanish Jews sold everything and hired a boat to take his family. The captain dumped them on some island, his whole family died, and he davened – Hashem, maybe you are trying to push me away, but I will not go. I am not saying that this quote contradicts the OP, but I presume that a person who can sell possessions on a cheap and still rent a boat would not be a peasant. And more generally, do not rely on short excerpts, the author wrote long books for a reason.