Paxton was born and raised in Fort Worth, Texas, the son of Mary Lou (née Gray) and John Lane Paxton. His father was a businessman, lumber wholesaler, museum executive, and occasional actor. His mother was Roman Catholic, and he and his siblings were raised in her faith. Paxton was in the crowd when President John F. Kennedy emerged from the Hotel Texas on the morning of his assassination on November 22, 1963. Photographs of an 8-year-old Paxton being lifted above the crowd are on display at the Sixth Floor Museum in Dallas, Texas. On the Marc Maron podcast, Paxton revealed that at the age of 13 he contracted rheumatic fever which had damaged his heart. During his teens, Paxton worked as a paper delivery boy with Mike Muir.