He was educated at University College, Oxford, and was Perpetual curate of Buxton until his appointment as the Bishop of Madras in 1837.
The post came through the influence of John Hobhouse, 1st Baron Broughton, his brother-in-law. His last post was as Rector of Walton on the Wolds, Leicestershire. Spencer married in 1823 Harriet Theodora, daughter of Sir Benjamin Hobhouse.
Bishop George Spencer"s brother Aubrey Spencer (1795–1872) became first Bishop of Newfoundland in 1839, the Bishop of Jamaica.