He was educated at the University of Alabama and the University of Virginia, where he was a member of the Delta Kappa Epsilon fraternity (Eta chapter).
He practiced law in Greenville until the Civil War. Herbert entered the Confederate Army as a second lieutenant. He served as captain of the Greenville Guards, and was later promoted to the rank of colonel of the Eighth Regiment, Alabama Infantry.
Herbert was wounded at the Battle of the Wilderness May 6, 1864.
After the war, Herbert returned to his law practice in Greenville, Alabama. He was elected to Congress in 1877 from Montgomery, Alabama as a Democrat.
He served eight terms in this office. During his tenure as Congressman, Herbert was chairman of the Committee on Naval Affairs and was largely responsible for the increased appropriations which led to the revival of the United States Navy.
However, he was among those who favored a more limited program than the one proposed by Secretary of the Navy Benjamin F. Tracy in 1890, which called for 40 battleships.
Only four battleships were authorized as a result. Tracy had been influenced by the works of naval strategist Captain Alfred Thayer Mahan, which called for a large fleet capable of offensive action. Herbert also became well known for leading a charge in Congress to reduce the funding of the United States Geological Survey, resulting in a public feud with paleontologist Othniel Charles Marsh.
In 1893, President Grover Cleveland appointed Herbert as Secretary of the Navy.
By this time Herbert was able to muster support for an enlarged navy, despite the Depression of 1893, and brought the fleet to some level of preparedness for the Spanish–American War. Having converted to Mahan"s school of naval thought, five battleships (the Kearsarge and Illinois classes) and sixteen torpedo boats were authorized during Herbert"s tenure.
From 1897, when he left his Cabinet seat, to his death on March 6, 1919, Herbert practiced law in Washington, District of Columbia
United States Ship Herbert (Doctor of Divinity-160) was named in honor of Secretary Herbert.
He also served as a member of the United States House of Representatives from Alabama.