There is a tradition that his paternal ancestors were of Italian origin, Sillimandi by name, but long domiciled in Switzerland, and that from there, by way of Holland, the first of the family came to America at the time of the great Puritan migration, settling eventually near what is now Fairfield, Connecticut In the records of Fairfield County he appears as Daniel Sillivant. By 1690 the name of the family had been changed to Silliman and its members were becoming people of substance and prominence. Benjamin's grandfather, Ebenezer Silliman, graduated from Yale College in 1727, as did his son, Benjamin's father, Gold Selleck Silliman, in 1752. Both followed the law as a profession. The former was a member of the Governor's Council and a judge of the superior court; the latter was a general in the Continental Army in charge of the defense of Connecticut against the British. On the side of his mother, Mary, daughter of Rev. Joseph and Rebecca (Peabody) Fish of North Stonington, Connecticut, Benjamin Silliman was descended from Puritan stock through the Peabodys of Rhode Island, who derived from John and Priscilla Alden. He was born in a part of what is now Trumbull, Connecticut, and brought up at Fairfield.