Mr. Brewer was born on June 20, 1837 in Smyrna, Ottoman Empire (now İzmir, Turkey), to Emilia Field Brewer and Rev. Josiah Brewer who at the time of his birth were running a school for Greeks in Smyrna, Ottoman Empire; Emilia Brewer's brother Stephen Johnson Field, a future Supreme Court colleague of Mr. Brewer, was living with the couple at the time. His parents returned to the United States in 1838 and settled in Connecticut.