Following a private education, he studied at Christ Church, Oxford (1727), taking an Master of Arts in 1734.
His first ecclesiastical appointment was as vicar of Street Mary, Blackburn (1738, resigned 1742), followed by rector of Elm, with Emneth Chapel (Norfolk, 1738–1755). In 1741 his father installed him as Archdeacon of Oxford. He obtained the degree of Bachelor's Degree the same year.
In 1742 he was collated by his father to the vicarage of Lydd (Kent), together with the Rectory of Chiddingstone.
He was installed as a prebendary of Canterbury Cathedral in 1745 (Stall XII). He obtained the degree of Doctor of Divinity in 1746, and the following year exchanged Chiddingstone for the rectory of Wrotham (Kent).
In 1766 he succeeded Doctor William Freind as Dean of Canterbury. He died in 1770 at Wrotham where he is buried.
Potter was disinherited by his father as a result of a marriage of which the archbishop disapproved but he nevertheless enjoyed considerable preferment within the church as a result of his father"s patronage.