Graduating from a military academy in Ossing, New York, he served as a second lieutenant in the 5th Massachusetts Colored Cavalry at 17 years old. After the Civil War, he remained in the Army assigned to the 25th Army Corps in Texas and the 8th United States Cavalry in California, Oregon and Arizona fighting in many Indian campaigns until 1879. From 1881-1903, he was employed by Rathbone, Sard & Company, stove manufacturers in Albany.
He served as brigadier general of the 3rd brigade of the New York State Militia.
In 1903, he was appointed Assistant Secretary of War by President Theodore Roosevelt and continued under President Taft, serving for 10 years. General Oliver spoke at the dedication of several monuments to Civil War Union Units from Pennsylvania at the Antietam Battlefield in 1904.
The content of his speech can be found in the reference. Robert Shaw Oliver married Marion Rathbone in 1870.
They had four children: John Rathbone, Marion Lucy, Elizabeth Shaw, and Cora Lyman Oliver.
John was a Harvard graduate, priest, scholar, and physician.