There are several points to consider:
#1 – Going to a Rebbe is something you can only understand, if you’ve been to one yourself. And I don’t mean going as a sightseer; I mean going like you mean it.
#2 – Wives and kids manage quite well without us, for a short period of time. Most current example? The scene in Sullivan County. Besides, its believed (at least by chassidim) that going to a rebbe (or in this case, a kever) fuels your entire year, so going away for 5-10 days does not hurt the wife/kids… it helps.
#3 – As far as leaving EY, that I annot comment on, as I live in chutz.
#4 – Money to the local Uman economy, there I agree with you. That is sinful, but unavoidable. But no worse than buying a german car, or hiring a polish goyta. Sad, but a fact of life.
#5 – Aniyim – They get the $$ they need anyways, so the money spent going to a rebbe does not affect the aniyim. And, many “aniyim” make the trip to the rebbe themselves, so the $$ spent by working people going to the rebbe is well within their rights. (as if they need to obtain society’s permission, which they don’t).
Lastly, go one time and you will understand. I have yet to make the trip to Uman myself, but I have been to places where davening is on a different plane than it is in my hometown, and there is no comparision. The people, the place, the time, all these things add up to an experience that you need to be in it to understand it.