Stevens was born on April 20, 1920, in Hyde Park, Chicago, Illinois, to a wealthy family. His paternal grandfather had formed an insurance company and held real estate in Chicago, while his granduncle owned the Chas A. Stevens department store. His father, Ernest James Stevens (1884-1972), was a lawyer who later became a hotelier, owning two hotels, the La Salle and the Stevens Hotel. He lost ownership of the hotels during the Great Depression and was convicted of embezzlement (the conviction was later overturned). (The Stevens Hotel was subsequently bought by Hilton Hotels and is today the Chicago Hilton and Towers.) His mother, Elizabeth Maude (Street) Stevens (1881-1979), was a high school English teacher. Two of his three older brothers also became lawyers.