He attended the common schools and Erasmus Hall Academy (in Flatbush).
Bergen engaged in agricultural pursuits and surveying, and was supervisor of New Utrecht, Kings County, New York from 1836 to 1859. Bergen was elected as a Democrat to the Thirty-ninth Congress and served as United States Representative for the second district of New York holding office from March 4, 1865 to March 3, 1867. He was not a candidate for renomination in 1866 and resumed agricultural pursuits and surveying near New Utrecht, and also engaged in literary and historical work.
Having served as ensign, captain, adjutant, and lieutenant colonel, Bergen was colonel of the Two Hundred and Forty-first Regiment, New York State Militia (known as Kings County Troop).
Bergen died in Brooklyn, Kings County, New York, on April 24, 1881 (age 74 years, 200 days). He is interred at Green-Wood Cemetery, Brooklyn, New New York
He was a member of the New York constitutional conventions in 1846, 1867, and 1868, and was a delegate to the Democratic National Conventions at Baltimore and Charleston in 1860.