The eldest son of John Wiffen, an ironmonger, by his wife Elizabeth Pattison, both from Quaker backgrounds, he was born at Woburn, Bedfordshire, on 30 December 1792. Benjamin Barron Wiffen was his younger brother, and his youngest sister Priscilla married Alaric Alexander Watts. His father died young, leaving six children to Elizabeth"s care.
At age 14, Wiffen was apprenticed to Isaac Payne, a schoolmaster at Epping, Essex. Wiffen declined the degree of Doctor of Laws from Aberdeen University in 1827.
At the age of ten Jeremiah entered Ackworth School in Yorkshire, where he acquired some skill in wood engraving. In 1811 he returned to Woburn and opened a school in Leighton Road. By hard study he made himself at home in the classics and Hebrew, French, and Italian, and later, Spanish and Welsh.
In the summer of 1821 he was appointed librarian at Woburn Abbey to John Russell, 6th Duke of Bedford.
His death was sudden, at Froxfield, near Woburn, on 2 May 1836. He was buried on 8 May in the Friends" graveyard, Woburn Sands, Buckinghamshire.
They had three daughters.
