Caraway was born on a farm near , the youngest of three children. His father, Tolbert Caraway, was a country doctor and Confederate veteran; his mother was Mary Ellen Caraway. When he was 6 months old, his father was assassinated in a feud, leaving his family impoverished. He worked as a farmhand from age 7, then later as a railroad section hand, a farm tenant, and as a share-cropper. He studied at night and attended the common schools as a boy.