His house is now a hotel and his Stonehenge a picnickers" spot on nearby Forestry Commission land. From 1763 to 1770 he had a private tutor at Eton College. On 24 October 1770 he was admitted as a fellow-commoner to Christ"s College, Cambridge.
In 1784 he served the office of High Sheriff of Yorkshire.
He was twice married: first in 1775 to Caroline (d 1821), daughter of Henry Seymour, and second on 5 January 1822 to Anne Holwell, second daughter of William Gater. Danby almost entirely rebuilt his country house at Swinton, from designs by John Carr and local builder-architects, with some interior design contributed by James Wyatt.
lieutenant included a handsome library and a richly furnished museum of minerals. Describing a tour which he made in 1829, the poet Robert Southey remarked, "The most interesting person whom I saw during this expedition was Mr.
Danby of Swinton Park, a man of very large fortune, and now very old."
Danby died at Swinton Park on 4 December 1833.
He left no children.