Rafelson was born to a Jewish family in New York City, the son of a hat ribbon manufacturer. His uncle was screenwriter and playwright Samson Raphaelson, the author of The Jazz Singer, who wrote nine films for director Ernst Lubitsch. "Samson took an interest in my work," Rafelson told critic David Thomson. "If he liked a picture, then I was his favorite nephew. But if he didn't like it, I was a distant cousin!" Rafelson had an older brother, Donald, and attended Trinity-Pawling School on scholarship.