Holder was born on January 21, 1951, in New York City to Barbadian immigrants Eric "Sonny" Holder Sr. and Miriam Holder. He was raised in a working-class section of Queens, New York, where his father, after a series of jobs, went into real estate, and his mother worked as a church secretary in order to keep a close eye on her two sons, Eric and William. In the fourth grade, Holder was identified as a gifted student and was transferred from his mostly black elementary school to a mostly white one. Describing his first day at that school he has said: "Everybody knew everybody, and nobody was a Negro. I was feeling awkward when this kid, Frank Sardelis, walked to me and put out his hand".