“total restraint is the same as eliminating one element of checks & balances from the federal government, especially during “progressive” era.”
Their restraint was a reaction to the excesses of the notorious “Lochner” era in which the Supreme Court repeatedly tossed out attempt after attempt to provide any sensible economic regulation at either the state or federal levels. Brandeis and Cardozo felt that it was appropriate for State and Federal Governments to take actions to make markets fairer (Brandeis had been a huge anti-monopoly advocate) and to relieve human misery during the Great Depression and that the courts shouldn’t be trying to legislate. But they even voted along with the majority in A.L.A. Schechter Poultry Corp. v. United States to rule the National Recovery Act unconstitutional.
Frankfurter didn’t join the Court in time for the New Deal cases. (He took Cardozo’s seat.) But he in general was quite hostile to having the courts overturn what legislatures had done, objecting for example to the Court trying to enforce religious freedom mandates. He would be horrified by the current Court.