The fourth of six children, James Gibbons was born in 1834 in Baltimore, Maryland, to Thomas and Bridget (née Walsh) Gibbons. His parents were from Tourmackeady, County Mayo, Ireland, and settled in the United States after moving to Canada.
After falling ill with tuberculosis in 1839, his father moved the family to his native Ireland, where he believed the air would benefit him. There, Thomas operated a grocery store in Ballinrobe. His father died in 1847, and his mother returned the family to the United States in 1853, settling in New Orleans, Louisiana.