Nobody ,Jew or non-Jew, likes changing demographics resulting in their group being a diminishing presence in the area and some other group that doesn’t share their interest or political values taking over.
That said it also depends on where and what the concern is. I understand the resentment in a place where the frum community is rapidly growing and demanding changes to the budget and zoning to accommodate them. In a large urban area where even if the frum community does take over one neighborhood they still won’t be a big enough voting presence to cause any changes to the lives of the people living just a few blocks away from them, the resentment is not justified.