Clarke's father was a Polish immigrant, and her mother was of Jewish descent.
Shirley Clarke was born on October 2, 1919, in New York City, New York, the United States, to a Polish-immigrant, a wealthy manufacturer Samuel L. Brimberg, and Florence Rosenberg Brimberg. Her mother was the daughter of a multimillionaire Jewish manufacturer and inventor. Her sister was the writer Elaine Dundy. Despite the privileges of their wealth, the Brimbergs’ home life was difficult because of Samuel's temper and physical abuse, and Shirley and her father often clashed. Her interest in dance began at an early age but met with the disapproval of her father, a violent bully.