Walter Edward Williams was born on March 31, 1936 in Philadelphia, Pennsylvania, United States. Williams's family during his childhood consisted of his mother Catherine Williams, a domestic worker; his sister, and him. His father deserted the family when Walter was three.
He grew up in Philadelphia. The family initially lived in West Philadelphia, moving to North Philadelphia when Williams was ten years old.
Williams is a cousin of former NBA player Julius Erving.