Perry was born in Boston, Massachusetts, United States, on January 13, 1848. She was a daughter of Hannah Lowell Jackson Cabot and Dr. Samuel Cabot III, a distinguished surgeon. She was the eldest of eight children, among whom were Samuel Cabot IV (born in 1850), chemist and founder of Valspar's Cabot Stains, Dr. Arthur Tracy Cabot (born in 1852), a progressive surgeon, Godfrey Lowell Cabot (born in 1861), founder of Cabot Corporation, James Jackson Cabot, Helen Jackson Cabot and Guy Chilton Cabot.
The Cabots were prominent in Boston society, and the circle of her family included Ralph Waldo Emerson, James Russell Lowell (her mother's cousin), and Louisa May Alcott. Perry recalled having the possibility to play the game called "fox and geese" with both Alcott and Emerson.