James Broadhead was born on May 29, 1819 at Charlottesville, Albemarle County, Virginia. His mother, Mary Winston (Carr) Broadhead was of Scottish origin, her ancestors occupying large estates in Virginia. His father, Achilles Broadhead, was a native of Albemarle County. A brother, Garland Carr Broadhead later attained eminence as a geologist. Achilles Broadhead was a man of great force of character, intensely patriotic, and a soldier of the War of 1812.