Too bad the OP has to stay in NJ or I would have suggested Queens and/or Far Rockaway. Cedarhurst or Inwood are the somewhat less pricey options in 5Towns.
Be that as it may, Edison/HP is the “frummer” side of Elizabeth. Definitely Passaic for its live and let live attitude. And Teaneck is moving a bit more to the right, but I wouldn’t call it “inexpensive”.