Their scales are an issue of great historical debate; Rav Machpood recently partook of a mesora seudah in Tel Aviv that served swordfish.
But it’s true, most Poskim tell us to stay away from them (only the little ones have scales, and they can’t be taken off).