Ethnicity: His mother was of Croatian, Anglo-Irish, and Scottish descent.
Elwes was born in Westminster, London. He is the third and youngest son of portrait-painter Dominick Elwes and interior designer Tessa Georgina Kennedy, who has Serbian, Croatian-Jewish, Anglo-Irish, and Scottish ancestry. His paternal grandfather was painter Simon Elwes, whose father was the diplomat and tenor Gervase Elwes (1866–1921). His brothers are Damian Elwes, an artist, and Cassian Elwes, a producer and agent. He was the stepson of Elliott Kastner, an American film producer. One of his ancestors is John Elwes, who is believed to be the inspiration for Ebenezer Scrooge in Charles Dickens' A Christmas Carol (1843) (Elwes played five roles in the 2009 film adaptation of the novel).
His parents divorced when he was four years old, and in 1975 when Elwes was thirteen his father committed suicide.