James was born on August 8, 1900, in Black Hawk County, Iowa, United States; the eldest of four children born to Katherine Hearst and Charles Hearst. His grandparents had settled on the farm in 1853, and his father was a farmer and a state leader in the field of agriculture, serving 13 years as president of the Iowa Farm Bureau Federation.
As a teenager, “Jim” worked hard on the family farm. He was known as an outstanding athlete. As a teenager, he played semipro baseball under an assumed name in northeast Iowa.