Al Jolson was born as Asa Yoelson in the Jewish village of Srednik now known as Seredžius, near Kaunas in Lithuania, then part of the Russian Empire. He was the fifth and youngest child of Moses Rubin Yoelson (1858 – December 23, 1945) and Nechama "Naomi" Cantor (1858 – February 6, 1895); his four siblings were Rose, Etta, another sister who died in infancy, and Hirsch (Harry). Jolson claimed not to know when he was born, and later chose to claim he was born on May 26, 1886. His one-time sister-in-law, Margaret Weatherwax (a sister of Ruby Keeler), claimed Jolson was the same age as their father, Ralph (who was born in 1881), and that Jolson was 46 when he married the 18-year-old Ruby in 1928.
In 1891, his father, who was qualified as a rabbi and cantor, moved to New York to secure a better future for his family. By 1894, Moses Yoelson could afford to pay the fare to bring Naomi and their four children to the U. S. By the time they arrived, he had found work as a cantor at Talmud Torah Congregation in the Southwest Waterfront neighborhood of Washington, D. C. , where the family was reunited.
Hard times hit the family when his mother, Naomi, died at 37 in early 1895. Following his mother's death, young Asa was in a state of withdrawal for seven months.