He held office as Comptroller of the Household in 1755-1756 and as a Lord of the Bedchamber from 1756-1767, having succeeded his father as Earl in 1756. He was Ambassador from the United Kingdom to Russia from 1762-1765 and Lord Lieutenant of Ireland from 1776-1780, when his Chief Secretary was Sir Richard Heron, Baronet In the latter role he had to concede free trade and, more importantly, the enactment of the Papists Acting 1778 which partially repealed the Penal laws and provided measures for the relief of Roman Catholics and Dissenters.
Lady Harriet, Marchioness of Lothian, who married William Kerr, 6th Marquess of Lothian was the mother of John Kerr, 7th Marquess of Lothian
Lady Caroline, who married William Assheton Harbord, 2nd Baron Suffield
Lady Sophia, who married Richard Edgcumbe, 2nd Earl of Mount Edgcumbe
Amelia Stewart, Viscountess Castlereagh, wife of the Foreign Secretary Robert Stewart, Viscount Castlereagh.
He was laid to rest in the family mausoleum at Blickling Hall, the family seat in Norfolk.
Royal Society; 11th Parliament of Great Britain. 10th Parliament of Great Britain]
He was Member of Parliament for Norwich from 1747-1756, having also been elected for Street Ives in 1747 but opting to sit for Norwich.