Grosvenor was born on October 28, 1875, in Constantinople, Ottoman Empire (now Istanbul, Turkey), the son of the Reverend Edwin Augustus Grosvenor, a professor of history at the American-sponsored Robert College in Constantinople, and Lilian Hovey Waters. The Grosvenor twins, Gilbert and Edwin, spent their childhood in Turkey. The boys absorbed an intense interest in history and archaeology from their father and even assisted him in the preparation of a two-volume history of the area. The work's numerous photographs stimulated Gilbert's interest in photography. In 1890 the Grosvenor family returned to the United States.