Anson was born on February 22, 1838 in New York City, New York, United States, where shortly before 1800 his grandfather, Thomas Stokes, coming from London, had settled as a merchant. He was the son of James Boulter and Caroline (Phelps) Stokes and brother of William Earl Dodge Stokes and Olivia Egleston Phelps Stokes. On his mother's side, he was the grandson of Anson Greene Phelps, and a descendant of George Phelps who emigrated from Gloucestershire, England, to Dorchester, Massachussets, about 1630.
His immediate ancestors were noted, not only for their business ability, but also for their religious, civic, and philanthropic interests. Thomas Stokes had been one of the thirteen founders of the London Missionary Society, and after coming to the United States was an active supporter of the American Bible Society, the American Tract Society, and the American Peace Society. James Boulter Stokes was one of the founders of the Association for Improving the Condition of the Poor, and a trustee of other charitable institutions.