He was the father of four children: Charles, born 1826, Henry, born 1834, Elihu, born 1841, and Julia. He became a minister in 1831. In 1840, he and his family moved to Wisconsin where he was a missionary among the Oneida tribe near Green Bay, and later near Fond du Lac.
He was a Methodist minister and one of the first settlers of the area that would become known as Appleton.
He was commissioned by Amos Adams Lawrence to establish a frontier school and raise funds to build the school that would be known as Lawrence University. He also assisted in writing the charter of the university and was one of the first trustees of the university.
Colman died February 7, 1895 in Fond du Lac.