He was born in 1921 in Honolulu, Hawaii, the son of William and Margaret Thurston Twigg-Smith (1895–1931). He is the 2xgreat-grandson of two pioneer missionary couples: Asa and Lucy Goodale Thurston and Lorrin Andrews and his wife. He is the grandson of Lorrin A. Thurston, who played a key role in the 1893 overthrow of the Hawaiian Kingdom.
His father William was an artist and a musician, who supported his family as an illustrator at the Hawaiian Sugar Planters' Association. Twigg-Smith grew up in the lower Nuʻuanu Valley on Bates Street, in a house his father built on his grandfather Lorrin's property.