Ray was born on September 23, 1930, in Albany, Georgia, United States. When Ray was an infant, his family moved from Albany, Georgia, where he was born, to the poor black community on the western side of Greenville, Florida. Ray was an only child of Bailey Robinson, a mechanic, and Aretha Williams, a sharecropper.
One of the most traumatic events of his childhood was witnessing the drowning death of his younger brother. Soon after his brother's death, Charles gradually began to lose his sight.
At age 10, his father died. At age 15, his mother passed away. Ray was an orphan. He was taken in by friends of his mother.