Bertie was the third son of Willoughby Bertie, 4th Earl of Abingdon and Charlotte Warren, a daughter of Peter Warren (Royal Navy officer) and a descendant of the Schuyler family, the Van Cortlandt family, and the Delancey family of British North America. His father died insolvent, having sold off much of the estates at Westbury, Wiltshire, but he retained control of the burgages and hence the disposal of the Parliamentary seats for Westbury.
The 4th Earl had sold the nomination of members there since 1786 to raise money. His trustees and successor continued the practice until 1810, when he sold the manor and control of the seats to Sir Manasseh Lopes. Lord Abingdon was Lord Lieutenant of Berkshire between 1826 and 1854 and High Steward of Abingdon.
They lived at Wytham Abbey in Berkshire (now Oxfordshire) and had eight children:
Montagu Bertie, 6th Earl of Abingdon (1808–1884)
Lady Charlotte Margaret Bertie (23 July 1809 – 7 November 1893)
Lady Emily Caroline Bertie (11 August 1810 – 18 March 1891), married Honorary
Review Charles Bathurst (d 1842), without issue
Honorary Albemarle Bertie (26 September 1811 – 4 February 1825)
Honorary
Review Henry William Bertie (16 September 1812 – 31 December 1894), vicar of Great Ilford
Lady Augusta Georgiana Bertie (14 April 1815 – 7 May 1815)
Honorary
Vere Peregrine Bertie (23 November 1817 – 21 March 1818)
Lieutenant Honorary Brownlow Charles Bertie (19 August 1819 – 30 December 1852).
