Cha was born on February 6, 1924, in Haining, China as the second of six youngsters from the academic Zha group of Haining. He is purportedly a relative of Zha Jizuo (1601–1676), a researcher who lived in the late Ming line and early Qing dynasty.
His father, Zha Shuqing, was arrested and executed by the Communist government for allegedly being a counterrevolutionary during the Campaign to Suppress Counterrevolutionaries in the early 1950s. Zha Shuqing was later posthumously declared innocent in the 1980s.
Cha has three brothers and two sisters. He is the second oldest among them. His brothers are Zha Liangjian, Zha Lianghao and Zha Liangyu. His sisters are Zha Liangxiu and Zha Liangxuan.