Goyim never swore to be peaceful. They swore to not ? subjugate us “too much”(yoser midai). The rishonim say that the shavuos on us are because of kefirah in bias hamoshiach, which means that they have absolutely no connection to what the goyim swore to.
The ramban holds that there is a mitzvah on individuals to live there. He does not ever say that there is a mitzvah to fight for eretz yisroel bzman hagalus. He brings the shevuos in his maamar al hageulah, in fact.
Even if they aren’t halacha(which they are), they represent an ideal and a perception of galus that chazal are teaching us about. To say that we’re better off in a situation chazal say would “allow our flesh to be hunted like animals” is borderline apikorsus.