He was born in New Orleans to George Walter Anderson, a grain broker, and Annette McConnell Anderson, member of a prominent New Orleans family, who had studied art at Newcomb College, where she had absorbed the ideals of the American Arts and Crafts movement.
Like Walter, Peter attended Saint John"s School in Manlius, New York until their schooling was interrupted by World War I and they enrolled in Isidore Newman School (then called Isidore Newman Manual Training School) in New Orleans.
Anderson was the eldest of three brothers, the others being Walter Inglis Anderson (1903–1965) and James McConnell "Mac" Anderson (1907–1998). In 1918, the Andersons purchased a large wooded tract of coastal land in Ocean Springs.
Devastated by Hurricane Katrina, the Shearwater workshop was rebuilt by Patrick Ashley, grandson of Peter Anderson, and other members of the family.
