Madison is said to have been named Dorothea for Dorothea Dandridge, afterward the second wife of Patrick Henry, but she is known to history as "Dolly. " The eldest daughter and apparently the third child of John and Mary (Coles) Payne, she was born in what is now Guilford County, North Carolina, where her Virginian parents were spending a year with an uncle.
Her paternal grandfather, John Payne, was an Englishman who settled in Goochland County, Virginia and married Anna Fleming of Scotch descent. Her maternal grandfather was William Coles of "Coles Hill, " Hanover County, Virginia, and formerly of Enniscorthy, County Wexford, Ireland.
Brought to Virginia by her parents, Dolly grew up at "Scotchtown, " in Hanover County. A member of a Quaker family, the little maid lived a restrained country life. Finding the atmosphere of Virginia uncongenial and desiring to provide better educational opportunities for his numerous children, John Payne set free his slaves and in the summer of 1783 removed to Philadelphia, where he engaged unsuccessfully in business and died in 1792.