Brett was born on September 5, 1946, in Feldafing, Starnberg, Germany. Brett's parents, Max (Mojsze Brajtsztajn) and Rose (née Szpindler), lived in Łódź, Poland, before the outbreak of World War II. During that war, they survived more than five years of Nazi control including being confined to the Łódź Ghetto, where they married, in occupied Poland, before being taken to the Auschwitz concentration camp where they were eventually separated. After the European theatre of war ended in May 1945, it took six months for the couple to find each other. Brett was born as Luba Brajsztajn (Germanicised as Lilijahne Breitstein) in 1946 in Feldafing displaced persons camp in Bavaria, Germany. In 1948 she emigrated to Australia with her parents. Brett lived in North Carlton, Melbourne from 1948 to 1989 and then in New York City.