He is of English, Welsh, and Scottish ancestry (some of his Anglo ancestors lived in India, Australia, and New Zealand).
Lloyd was born on November 17, 1983, in London, the son of Marion Evelyn (née Dickens), a children's publisher, and Jonathan Lloyd, who heads a literary agency. He is the great-great-great-grandson of Victorian writer Charles Dickens through his mother, who is the daughter of Captain Peter Dickens, RN. One of his maternal great-grandfathers was Rear-Admiral Henry Blagrove. He is a cousin of biographer and writer Lucinda Hawksley, and actor and performer Gerald Dickens.