Schwartz was born Avram Moishe Schwartz in Sudlekov, in Austrian Galicia, to Isaac, a grain dealer, and his wife Rose Schwartz, a Jewish family. Moishe was the oldest of three boys among the six siblings, and had three older sisters. Like many similar families, the Schwartzes immigrated to the United States in stages. In 1898 Isaac Schwartz emigrated with his three teen-aged daughters, so they could all work to get started in New York and earn money for passage for Rose and their three young sons. The following year he sent tickets for his wife and the boys. They got as far as Liverpool, where they were to sail for the US, but got separated and Rose was forced to leave without Moishe. Without any English, he made his way to London, where he lived for two years, surviving with the help of strangers. His father located him in 1901, and they traveled together to New York when Moishe was twelve.