In his younger years, Harlan attended The Latin School of Chicago. He later attended two boarding high schools in the Toronto Area, Canada: Upper Canada College and Appleby College.
Upon graduation from Appleby, Harlan returned to the U.S. and in 1916 enrolled at Princeton University. There, he was a member of the Ivy Club, served as an editor of The Daily Princetonian, and was class president during his junior and senior years. After graduating from the university in 1920 with a Artium Baccalaureus degree, he received a Rhodes Scholarship, which he used to attend Balliol College, Oxford. He studied jurisprudence at Oxford for three years, returning from England in 1923.
Upon his return to the United States, he began work with the law firm of Root, Clark, Buckner & Howland (now known as Dewey & LeBoeuf), one of the leading law firms in the country, while studying law at New York Law School. He received his Bachelor of Laws in 1924 and earned admission to the bar in 1925. Between 1925 and 1927, Harlan served as Assistant United States Attorney for the Southern District of New York, heading the district's Prohibition unit. He prosecuted Harry M. Daugherty, former United States Attorney General. In 1928, he was appointed Special Assistant Attorney General of New York, in which capacity he investigated a scandal involving sewer construction in Queens. He prosecuted Maurice E. Connolly, the Queens borough president, for his involvement in the affair. In 1930, Harlan returned to his old law firm, becoming a partner one year later. At the firm, he served as chief assistant for senior partner Emory Buckner and followed him into public service when Buckner was appointed United States Attorney for the Southern District of New York. After Buckner's death, Harlan became the leading trial lawyer at the firm. In the following years Harlan specialized in corporate law dealing.
In 1937, Harlan was one of five founders of a eugenics advocacy group the Pioneer Fund, which had been formed to introduce Nazi ideas on eugenics to the United States. He had likely been invited into the group due to his expertise in non-profit organizations. Harlan served on the Pioneer Fund's board until 1954. He did not play a significant role in the fund.
During World War II, Harlan volunteered for military duty, serving as a colonel in the United States Army Air Force from 1943 to 1945. He was the chief of the Operational Analysis Section of the Eighth Air Force in England. He won the Legion of Merit from the United States, and the Croix de guerre from both France and Belgium. In 1946 Harlan returned to private law practice representing Du Pont family members against a federal antitrust lawsuit. In 1951, however, he returned to public service, serving as Chief Counsel to the New York State Crime Commission, where he investigated the relationship between organized crime and the state government as well as illegal gambling activities in New York and other areas. During this period Harlan also served as a committee chairman of the Association of the Bar of the City of New York, and to which he was later elected vice president. Harlan's main specialization at that time was corporate and anti-trust law.
Harlan was nominated by President Eisenhower on January 10, 1955, to a seat on the Supreme Court of the United States vacated by Associate Justice Robert H. Jackson. He was confirmed by the Senate on March 16, 1955, and received commission the next day. Despite the brevity of his stay on the Second Circuit, Harlan would serve as the Circuit Justice responsible for the Second Circuit throughout his Supreme Court capacity. Additionally, he served as Circuit Justice for the Ninth Circuit from June 25, 1963 to June 26, 1963. He assumed retired status on September 23, 1971, serving in that capacity until his death on December 29, 1971.
John Harlan held precedent to be of great importance, adhering to the principle of stare decisis more closely than many of his Supreme Court colleagues. He eschewed strict textualism. While he believed that the original intention of the Framers should play an important part in constitutional adjudication, he also held that broad phrases like "liberty" in the Due Process Clause could be given an evolving interpretation. Harlan believed that most problems should be solved by the political process, and that the judiciary should play only a limited role.
Harlan, a Presbyterian, maintained a New York City apartment, a summer home in Weston, Connecticut and a fishing camp in Murray Bay, Quebec, a lifestyle he described as "awfully tame and correct".
The justice played golf, favored tweeds, and wore a gold watch which had belonged to the first Justice Harlan. In addition to carrying his grandfather's watch, when he joined the Supreme Court he used the same furniture which had furnished his grandfather's chambers.
In 1928, Harlan married Ethel Andrews, who was the daughter of Yale history professor Charles McLean Andrews. This was the second marriage for her. Ethel was originally married to a New York architect Henry K. Murphy, who was twenty years her elder. After Ethel divorced Murphy in 1927, her brother John invited her to a Christmas party at Root, Clark, Buckner & Howland, where she was introduced to John Harlan. They saw each other regularly afterwards and married on November 10, 1928 in Farmington, Connecticut. John and Ethel Harlan had one daughter Evangeline Dillingham (born on February 2, 1932).