Gregory was born on November 6, 1846, in Philadelphia, Pennsylvania, of French extraction. His great-grandfather, René Grégoire, was a French officer, a Protestant, who came to America to serve under Washington in the Revolution. Having married in Santo Domingo and become a planter there, he was killed in an insurrection about 1797, and his young son, Caspar Ramsay Grégoire, was sent to Philadelphia, where he became a ship-captain and ship-owner, and Anglicized his name. The latter's son, Henry Duval Gregory, long the head of a classical school in Philadelphia and later vice-president of Girard College, was the father of Caspar René Gregory. His mother's name was Mary Jones.