He was the son of the 1st earl (peerage of the United Kingdom), who died in 1756. The English family of Fitzwilliam claimed descent from a natural son of William the Conqueror, and among its earlier members were a Sir William Fitzwilliam (1460 - 1534), sheriff of London, who in 1506 acquired the family seat of Milton Manor in Northamptonshire, and his grandson Sir William Fitzwilliam. The latter's grandson was made an Irish baron in 1620; and in later generations the Irish titles of Viscount Milton and Earl Fitzwilliam (1716) and the English titles of Baron Milton (1742) and Viscount Milton and Earl Fitzwilliam (1746), were added. These were all in the English house of the Fitzwilliams of Milton Manor. They were distinct from the Irish Fitzwilliams of Meryon, who descended from a member of the English family who went to Ireland with Prince John at the end of the 12th century, and whose titles of Baron and Viscount Fitzwilliam died out with the 8th viscount in 1833; the best known of these was Richard, 7th viscount (1745 - 1816), who left the Fitzwilliam library and a fund for creating the Fitzwilliam Museum to Cambridge University. The 2nd earl inherited not only the Fitzwilliam estates in Northamptonshire, but also, on the death of his uncle the marquess of Rockingham in 1782, the valuable Wentworth estates in Yorkshire, and thus became one of the wealthiest noblemen of the day.