Taubman was born in Pontiac, Michigan on January 31, 1924 to Jewish immigrants Fannie Ester Blustin and Philip Taubman.
His parents came to the United States from Białystok, in northeastern Poland. His mother was his father's second cousin. Philip took a job with the Wilson Foundry Company in Iowa, transferred to Pontiac in 1920, became a fruit farmer, then began developing commercial real estate and custom homes.
Taubman's parents lost everything in the Depression of the 1930s, and Taubman at age 9 had to find work to help support the family.