Charley's father was a black coal miner and his mother a white Irish immigrant from County Cork.
Charles Duane Burley was born in Bessemer, United States, on September 6th 1917. When the hard, physical labour of the mines claimed his father in 1925 Charley and his family moved to Pittsburgh.
At age 12, Charley joined the Kay Boys Club where he took up boxing under the watchful eye of local trainers Leonard Payne and Howard Turner. Charley enjoyed the boxing as much as he enjoyed baseball, another sport at which he excelled, (it has been said that he once received an offer to play for the Homestead Grays), and when he wasn't playing ball or plucking chickens for pennies, (a skill he learned in Bessemer), he could be found at the gym.