On August 29, 1815, Anna Ella Carroll was born in a lavish, twenty-two-room manor called Kingston Hall, which rested on a large Maryland plantation stocked with cotton, wheat, and tobacco. Anna was a bright, blue-eyed baby with dark red curls and a fair complexion. She had in girlhood a fierce temper and an independent spirit, balanced with an equally strong tendency to shower her family with love. Her sense of independence would remain with her, carrying her through the adventures that lay ahead.
For generations, the Carrolls had been an influential family in America. Thomas King Carroll and Juliana Stevenson Carroll, Anna's parents, were extremely wealthy and well-respected people of the South. As a teenager, Juliana had been an accomplished organist for the Episcopal church. Thomas Carroll was a powerful lawyer whose partners included Francis Scott Key, the composer of America's national anthem. Anna Carroll's paternal grandfather, Charles Carroll, signed the Declaration of Independence. Her maternal grandfather, Doctor Henry Stevenson, served as an officer and a surgeon in the British navy during the Revolutionary War. He operated on Tory soldiers and American prisoners of war alike, earning the respect of men on both sides.
Anna Carroll was the first of eight children, only two of whom were boys. She soon became Anne to her family and friends, rarely using her real birthname, even in adulthood. Anne led a privileged life as a child, with a slave caretaker, Milly, to care for her every need from the time she was born. She also had a personal servant, a beautiful slave girl her own age named Leah, who tended her for many years. Anne and Leah became friends, yet they always observed the boundaries of their positions as mistress and servant.