John Graham represented the Nelson electorate in the House of Representatives for 18 years from 1893 to 1911, when he retired. The 1893 general election was contested by four candidates in Nelson: Graham (1289 votes), Richmond Hursthouse (1011 votes), John Kerr (910 votes) and William Lock (74 votes). The 1896 general election was contested by Graham and Jesse Piper, who received 2061 and 1718 votes, respectively.
Graham was thus elected with a majority of 343 votes.
In 1894, Graham successfully opposed the Midland Railway Bill insofar as it proposed to abandon the obligations of the company to provide a railway to Nelson. As Chairman of the Banking Enquiry Committee of 1896, John Graham frequently came into conflict with the Premier Richard Seddon, the Minister of Lands, John McKenzie, and George Hutchison
Graham was for some years chairman of the Town Schools" Committee.
He was the first chairman of the Nelson Harbour Board and held that position for twelve years. He served on the Nelson Education Board, the Board of Governors of Nelson College (1894) and Council of Victoria College in Wellington (1899–1912).
Graham died in Feilding on 8 February 1926, after having lived in Nelson for practically all of his life.
He was also a member of the Nelson City Council.