Burton Stephen Lancaster was born on November 2, 1913, in Manhattan, New York, at his parents' home at 209 East 106th Street. He was the son of Elizabeth (née Roberts) and mailman James Henry Lancaster. Both of his parents were Protestants of working-class origin.
All four of his grandparents were Irish immigrants to the United States, from the province of Ulster; his maternal grandparents were from Belfast and were descendants of English immigrants to Ireland.
They resided in Italian East Harlem. When Lancaster and his brothers were old enough, they shoveled snow, sold newspapers, and shined shoes to earn money for the family. While James Lancaster was a gentle, warm father, Elizabeth was a strict disciplinarian, instilling in her children the virtues of honesty and loyalty, with whippings if necessary. She had no prejudices against the many different ethnic groups in her neighborhood and treated them all kindly, which made a strong impression on her son.
Lancaster's mother died of chronic intestinal nephritis.
In September 1961, Lancaster's father died. James Lancaster had lived with his son since 1947. The two had been very close.