He attended the public schools in Barnet and graduated from Saint Johnsbury Academy in 1876 and Dartmouth College in Hanover, New Hampshire in 1880. He studied law and was admitted to the bar in 1883.
He served as a United States. Representative from Vermont. He began the practice of law in Burlington, Vermont. Foster served as Chittenden County State"s Attorney from 1886 until 1890.
Foster was the first president of the Young Men"s Republican Club of Vermont, which was organized in 1894.
He was commissioner of State taxes from 1894 until 1898. He served as chairman of the board of railroad commissioners from 1898 until 1900, and as chairman of the commission representing the United States at the first Centennial of the Independence of Mexico at Mexico City in 1910.
Foster was the chairman of the United States delegation to the general assembly of the International Institute of Agriculture at Rome in May 1911. Foster was elected as a Republican candidate to the Fifty-seventh and to the five succeeding Congresses, serving from March 4, 1901 until his death in Washington, District of Columbia on March 21, 1912.
He served as chairman of the Committee on Expenditures in the Department of Commerce and Labor during the Fifty-ninth, Sixtieth and Sixty-first Congresses.
He served as the chairman on the Committee on Foreign Affairs in the Sixty-first Congress. Foster was interred in Lakeview Cemetery in Burlington, Vermont.
He served as a member of the Vermont State Senate from 1892 until 1894.