Van Doren was born on March 16, 1891, in Birmingham, Alabama. She was the daughter of John Taylor Bradford, a lumberman and merchant, and Ida Henley Brooks. When she was four, her family moved to Tallahassee, Florida, where her father owned a sawmill.
Five years later, she was shot and killed by a discharged worker, and Irita's mother had to support the family by giving music lessons and selling preserves and jellies. All four children had a strong interest in literature, and while helping their mother put up jars of preserves, they took turns reading aloud from the classics.