William Hallowes Miller was born on April 6, 1801, at Velindre near Llandovery, Carmarthenshire, South Wales. His father, Captain Francis Miller, who served in the American war, had a long military ancestry. By his first wife, Captain Miller had three sons, all of whom entered the army, and two daughters. After losing his estate near Boston, Massachusetts, he retired to Wales to the small estate of Velindre near Llandovery and in 1800 married Ann Davies, the daughter of a Welsh vicar. William was the only child of this second marriage; his mother died a few days after his birth, but his father lived to the age of eighty-six, dying in 1820.