Smith joined West Bromwich Albion as an amateur, but left to join Torquay United in 1953 without breaking into the Baggies" first team He initially lost his place when Drinkwater returned, but after two games out of the side, regained his place, Drinkwater switching to right-back as the on-loan Harry Parfitt missed out. He lost his place late in the season, with John Anderton taking his place and played just 14 times in the 1955-1956 season.
He started the 1956-1957 season out of the first team with Anderton and then regular centre-half John James playing at left-back.
However, Smith soon regained his place and remained a regular for the remainder of the season as Torquay lost out on promotion to Division Two on goal average from Ipswich Town. He played 188 league games for Torquay, scoring just once, before leaving in July 1961 to join Bristol City.
He played just once for the Ashton Gate side before leaving league football to settle down in Torquay, where he became an avid pigeon racer.