Eugene Walter was born on November 30, 1921, in Mobile, Alabama, United States to the family of Eugene Ferdinand Walter Sr. and Muriel Selina Olson. According to his memoirs, Walter ran away from home at the age of three and was raised by his paternal grandparents, who inspired his love of food and language. His Austrian grandmother, who spoke French when she wanted to share gossip, inspired his desire to live abroad. He became acquainted with Truman Capote, who was known as Bulldog Persons, when both were children.
At the deaths of his grandparents, he was informally adopted by Hammond Gayfer, who was the heir to Gayfer's Department Stores. Gayfer died in 1938, again leaving Walter to fend for himself. He later told friends and acquaintances that he had lived in the back of a bookstore at the age of 10. His hometown, which he described as unique in American society for its multicultural flavor, always loomed large in his storytelling. In adolescence, he became affiliated with the Children's Theatre Guild and wrote and performed marionette shows.