After starting as an apprentice to a linen draper in Leven, and working as a clerk in Dundee, he entered the Wesleyan theological college at Hoxton, and in 1804 was appointed minister of the first Scottish Congregational chapel in Aberdeen. In 1818 Philip joined the delegation headed by Rev. John Campbell to investigate the threatened closure of London Missionary Society's stations in and reported that the conduct of the Cape Colonists towards the indigenous people was deserving of strong reprobation.
In 1822 Philip was appointed superintendent of the London Missionary Society's stations in It was the period of the agitation for the abolition of slavery in England, where Philip's charges against the colonists and the colonial government found powerful support. In 1823 he went back to England to lobby for the indigenous and coloured people's civil rights. His recommendations were adopted by the House of Commons, but his unpopularity in grew.
The British government, however, forced the Cape government to conform to the views of Philip and the ordinance of 1828 was passed granting all free coloured persons at the Cape every right to which any other British subjects were entitled. The French Capetonian actor, polyglot and playwright Charles Etienne Boniface however produced a play in Dutch against Philip: "De nieuwe ridderorde of De Temperantisten". It was printed in 1832 (repr 1954).
In 1834, Sir Benjamin d'Urban became governor and was anxious to promote the interests of the indigenous people. D'Urban was dismissed by Charles Grant, 1st Baron Glenelg, the colonial secretary on the 1 May 1837. In 1849 Philip severed his connection with politics after the annexation of the Griqua lands and retired to the mission station at Hankey, Cape Colony, where he died in 1851.
The town of Philippolis in the Free State province is named after John Philip. The Transgariep Museum, in the town, has a section devoted to John Philip.