Berry was born in Jackson County, Alabama, on May 15, 1841. His parents, James M. and Isabelle (Orr) Berry, were farmers, and ten of their children lived to adulthood: Granville, Mary, Fannie, Dick, James, Arkansas, Willie, Sophrona, Albert, and Emma.
In 1848, his family moved to Arkansas, settling on a farm near Carrollton (Carroll County), where Berry’s father opened a store. Berry “learned to read and write a little, and something of arithmetic” during occasional school sessions, but mostly he helped on the farm. When Berry was seventeen, his father enrolled him in the Berryville Academy. Ten months later, his mother died, and Berry’s father sold the farm. Berry went to Yellville (Marion County) to work in the store of a cousin also named James H. Berry.