Antonia Pantoja was born to Alejandrina Pantoja on September 13, 1921, in Puerta de Tierra, Puerto Rico, a poor section of San Juan. She never knew her father who was not married to her mother. Toni, as she was affectionately called, witnessed the struggles of her family and friends organizing for better wages and living conditions. Antonia Pantoja Acosta was raised by her maternal grandparents Conrado Pantoja Santos and Luisa Acosta Rivera in the working-class neighborhood of Barrio Obrero.
Her grandfather, a cigar factory worker at the American Tobacco Company, was also an active union leader whose organizing efforts led to one of the first successful labor strikes in Puerto Rican history, notwithstanding the fierce and violent confrontations with the police and strikebreakers that ensued. After the workers won the strike, the company packed up and left Puerto Rico. She reported that being an eyewitness to this struggle impacted her life: "My life's work has always been influenced by the memory of this inequity."