Charles K. Kao was born on November 4, 1933, in Shanghai, Republic of China, to Kao Chun-Hsiang and King Tsing Fong. His father, an alumnus of Michigan Law School, was a successful Shanghai attorney. Later, he became a legal adviser and an instructor in Chinese law in Hong Kong.
Kao had a younger brother called Timothy. He also had two elder siblings, a sister and a brother, who died of measles before he was born. Kao held dual citizenship in Great Britain and the United States.