His mother was born in Casablanca, Morocco, to a Danish father and a French mother.
Asbæk was born on March 2, 1982, in Copenhagen, the son of gallery owners Maria Patricia (née Tonn) and Jacob A. Asbæk, who run Galerie Asbæk in Copenhagen. His father is from Hammel. He has two older brothers, Thomas Asbæk, an art consultant at Asbæk Art Consulting, and Martin Asbæk, a gallery owner at Martin Asbæk Gallery.
The nickname Pilou came from his mother's French ancestry, and is a French version of the word Pip. The nickname is because Asbæk was the youngest son, in French, "petit" for littlest, "Little Philip." Pilou is an abbreviation of Petit Philip.