Rice was born Elmer Leopold Reizenstein at 127 East 90th Street in New York City, New York. His grandfather was a political activist in the Revolutions of 1848 in the German states. After the failure of that political upheaval, he emigrated to the United States where he became a businessman. He spent most of his retirement years living with the Rice family and developed a close relationship with his grandson Elmer, who became a politically motivated writer and shared his grandfather's liberal and pacifist politics. A staunch atheist, his grandfather may also have influenced Elmer in his feelings about religion as he refused to attend Hebrew school or to have a bar mitzvah. In contrast, Rice's relationship with father was very distant. As he wrote in his autobiography, his grandfather and his Uncle Will, both of whom boarded with the family, made up for the affection and attention his father withheld. A child of the tenements, Rice spent much of his youth reading, to his family's consternation, and later observed, "Nothing in my life has been more helpful than the simple act of joining the library."