While there are varying accounts of her background, Gardner's only documented ancestry was English.
Gardner was born near the farming community of Grabtown, North Carolina. She was the youngest of seven children. She had two older brothers, Raymond and Melvin, and four older sisters, Beatrice, Elsie Mae, Inez, and Myra. Her parents, Mary Elizabeth "Molly" (née Baker) and Jonas Bailey Gardner, were poor cotton and tobacco farmers.
While the children were still young, the Gardners lost their property, forcing Jonas Gardner to work at a sawmill and Molly to begin working as a cook and housekeeper at a dormitory for teachers at the nearby Brogden School. When Gardner was seven years old, the family decided to try their luck in a larger city, Newport News, Virginia, where Mollie Gardner found work managing a boarding house for the city's many shipworkers. While in Newport News, Gardner's father became ill, and died from bronchitis in 1938, when Ava was 15 years old.
After Jonas Gardner's death, the family moved to Rock Ridge near Wilson, North Carolina, where Mollie Gardner ran another boarding house for teachers.