Stephen Van Rensselaer was born at New York City on the 1st of November 1764. He was fifth in descent from Killian Van Rensselaer, the original patroon of Rensselaerwyck, New York, who acquired his large estates between 1630 and 1637. His family was very wealthy, and the Van Rensselaer Manor House was a rich childhood environment for the young boy to grow up in. However, his father died in 1769 when Van Rensselaer was only five.
Van Rensselaer was raised by his mother and stepfather, the Rev. Eilardus Westerlo, whom his mother married in 1775, and his Livingston grandfather.