His early years and education remain obscure. His earliest recorded large commission was in his capacity as a builder and timber merchant at Thrigby Hall in 1735. He bought the freedom of Norwich in 1745 and was appointed carpenter the Great Hospital in the city in 1751.
In May 1750 he had put his house in the parish of Street Martin-at-Oak for sale.
He then leased a piece of land on the west forecourt of the Great Hospital on which he built a substantial residence for himself, into which he moved in 1756. lieutenant is now known as Street Helen"s House.
He went on to build the Assembly House (1754), the neighbouring theatre (1757, since destroyed), the Methodist meeting-house in Bishopsgate Street (1752-1753), the Octagon Chapel in Colegate (1754-1756) and the artillery barracks (1771). He was also responsible for various houses in the city.
In addition to work as a builder and architect, Ivory carried on an extensive trade importing timber, with his own yard in Bishopsgate.
He also operated the Norwich Theatre, and is recorded as having obtained a licence for his company of actors, the Norwich Company of Comedians, to perform in Norwich in 1768. In his will he is described as ‘builder and timber merchant. Ivory married Hannah Lacey (d1787) on 22 December 1735.
They had two sons: William, who assisted his father as an architect, and died in King Edward VI Almshouses, Saffron Walden, on 11 December 1837, aged 90, and Thomas, who emigrated to India, where he worked in the revenue office at Fort William.
They also had a daughter, Sarah.