His father was of English ancestry and his mother was born in Ireland.
He was born in Washington, D.C., on May 19, 1870, to Randall and Ellen Fish. He said that he was named after statesman and politician Hamilton Fish, a distant relative. His father was 43 years older than his mother and 75 years old at the time of his birth. Fish was the youngest child and had three living siblings: Walter, Annie, and Edwin. He wished to be known as "Albert" to escape the nickname "Ham & Eggs" that he was given at an orphanage in which he spent much of his childhood.
His family had a history of mental illness. His uncle suffered from religious mania. A brother was confined in a state mental hospital. His sister was diagnosed with a "mental affliction". Three other relatives were diagnosed with mental illnesses and his mother had "aural and/or visual hallucinations". His father was a river boat captain and by 1870 was a fertilizer manufacturer. The elder Fish died in 1875 of a myocardial infarction. Fish's mother then put him into Saint John's Orphanage in Washington, where he was frequently treated sadistically. He began to enjoy the physical pain that the beatings brought. Of his time at the orphanage, Fish remarked, "I was there till I was nearly nine, and that's where I got started wrong. We were unmercifully whipped. I saw boys doing many things they should not have done."
By 1880, his mother had a government job and was able to remove Fish from the orphanage. In 1882, at age 12, he began a relationship with a telegraph boy. The youth introduced Fish to such practices as urolagnia (drinking urine) and coprophagia (eating feces). Fish began visiting public baths where he could watch other boys undress, and spent a great portion of his weekends on these visits. Throughout his life he would write obscene letters to women whose names he acquired from classified advertising and matrimonial agencies.
By 1890, Fish arrived in New York City, and he said that at that point he became a prostitute and he also began raping young boys. In 1898, his mother arranged a marriage for him with a woman nine years younger than himself. They had six children: Albert, Anna, Gertrude, Eugene, John, and Henry Fish.