Hermite was born in Dieuze, The Moselle on 24 December 1822, with a deformity in his right foot which would affect his gait for the rest of his life.
He was the sixth of seven children of Ferdinand Hermite, and his wife Madeleine Lallemand. His father worked in his mother's family drapery business, and also pursued a career as an artist.
The drapery business relocated to Nancy in 1828 and so did the family. Hermite was the sixth of the seven children of Ferdinand Hermite and the former Madeleine Lallemand.