He served as the military secretary to Charles Seymour Whitman, the New York Governor. She would later become a missionary in the Philippines. They divorced in March 1922.
Together they had one child, a son named Lorillard Suydam Spencer, Junior.
On December 16, 1910 he was elected as the temporary chairman of the New York City Council of the Boy Scouts of America.
During World War I, he was the commander of the 3rd Battalion of the 15th New York Infantry. In September 1918, during the Battle of Champagne-Argonne, he was stuck six times by German machine gun fire.
Severely wounded, he was sent home to recover. The latter was presented to him in July 1919 by Ambassador Jusserand.
When the men of the 3rd Battalion returned home after the war, he was there to greet them, his fractured leg still in a sling.
They had two sons, Stephen Wolcott Spencer and William Hurlbut Force Spencer. Her sister Madeleine was the widow of Colonel John Jacob Astor IV. He died on June 10, 1939 in Newport, Rhode Island. He was buried in Green-Wood Cemetery in Brooklyn, New New York
The earliest Filipino Boy Scout troop recorded in history was named after him due to the monetary support provided by his mother, who was an American charity worker in Sulu, Philippines.
The Lorillard Spencer Trophy serves as the perpetual award to a local Boy Scout council in the Philippines obtaining the highest record of advancement and membership for the year. The trophy is inscribed as "The Lorillard Spencer Trophy Presented by His Mother in Loving Memory of a Loyal Scouter".