Smith was born on May 16, 1824 in St. Augustine, Florida, as the youngest child of Joseph Lee Smith, an attorney, and Frances Kirby Smith. Both his parents were natives of Litchfield, Connecticut, where their older children were born. The family moved to Florida in 1821, as the senior Smith was appointed as a Superior Court judge in the new Florida Territory, acquired by the US from Spain. Older siblings included Ephraim, born in 1807; and sisters Frances, born in 1809, and Josephine, who died in 1835, likely of tuberculosis. In 1836, Smith's parents sent their second son to a military boarding school in Virginia, which he attended until his enrollment in the United States Military Academy at West Point, New York.
The next year his sister Frances married Lucien Bonaparte Webster, a West Point graduate from Vermont and career Army officer, whom she met when he was stationed at Fort Marion in St. Augustine. His commanding officer was the Smiths' uncle. Webster later served in the Mexican-American War and died of yellow fever in 1853, when stationed on the Texas frontier at Fort Brown.