Charles Benedict Calvert was born on August 23, 1808; the grandson of Benedict Calvert, earlier known as Benedict Swingate. The latter, of uncertain maternity, was the son of Charles, fifth Lord Baltimore. Benedict's son George Calvert, removed from Mt. Airy to Riverdale, Maryland, occupying there a large estate which still remains in the family. He married Rosalie Eugenia Stier, of Belgian parentage, who became the mother of George Henry Calvert and Charles Benedict Calvert.