Wise was born in Budapest in the Austro-Hungarian Empire, the son and grandson of rabbis. His grandfather, Joseph Hirsch Weiss, was Chief Rabbi of a small town near Budapest. His father, Aaron Wise, earned a PhD and ordination in Europe, and emigrated to the United States to serve as rabbi of Congregation Baith Israel Anshei Emes in Brooklyn, New York. Wise's maternal grandfather, Móric Fischer de Farkasházy, created the Herend Porcelain Company. When Wise's father Aaron Wise sought to unionize the company, Moric gave the family one-way tickets to New York.
Wise immigrated to New York as an infant with his family. His father became rabbi of Rodeph Sholom, a Manhattan Conservative congregation of wealthy German Jews.