Nicholas, named after his paternal grandfather the emperor, was born as the eldest son to Grand Duke Nicholas Nicolaevich of Russia (1831–1891) and Alexandra Petrovna of Oldenburg (1838–1900) on 18 November 1856. His father was the sixth child and third son born to Nicholas I of Russia and his Empress consort Alexandra Fedorovna of Prussia (1798–1860). Alexandra Fedorovna was a daughter of Frederick William III of Prussia and Louise of Mecklenburg-Strelitz.
Nicholas' mother, his father's first cousin's daughter, was a daughter of Duke Konstantin Peter of Oldenburg (1812–1881) and Princess Therese of Nassau (1815–1871). His maternal grandfather was a son of Duke George of Oldenburg and Grand Duchess Catherine Pavlovna of Russia, daughter of Paul I of Russia and Maria Fedorovna of Württemberg. (Catherine was later remarried to William I of Württemberg.) His maternal grandmother was a daughter of Wilhelm, Duke of Nassau (1792–1839) and Princess Luise of Saxe-Hildburghausen. The Duke of Nassau was a son of Friedrich Wilhelm, Duke of Nassau (1768–1816) and Burgravine Louise Isabelle of Kirchberg. His paternal grandparents were Duke Karl Christian of Nassau-Weilburg (1735–1788) and Carolina of Orange-Nassau. Carolina was a daughter of William IV of Orange and Anne, Princess Royal and Princess of Orange. Anne was the eldest daughter of George II of Great Britain and Caroline of Ansbach.
Grand Duke Nicholas was the first cousin once removed of Tsar Nicholas II. To distinguish between them the Grand Duke was often known within the Imperial family as "Nikolasha"; the Grand Duke was also known as "Nicholas the Tall" while the Tsar was "Nicholas the Short".