James Harvey Robinson born in Bloomington, Illinois, on 29 June 1863, the son of James Harvey and Latricia Maria (Drake) Robinson. He was the third son and fifth of six children who survived infancy; two others died at an early age. His younger brother, Benjamin Lincoln Robinson, became a distinguished botanist and head of the Gray Herbarium at Harvard. On both his father's and mother's side Robinson's family line follows the characteristic course of the expansion of New England through upper New York state to the Old Northwest Territory. He was seventh in lineal descent from Isaac Robinson of Plymouth, son of the famous Dr. John Robinson, pastor of the Pilgrims at Leyden in Holland.
His father was born near Saratoga, New York, and settled in Bloomington in 1836, where he advanced from keeping a general store to a bank presidency.
His mother came to Bloomington with her father, a clergyman and small newspaper publisher, from Homer, New York, two years before her marriage in 1842.