Carey was born on March 4, 1526 in England, the second child of William Carey and Mary Boleyn who was the sister of Anne Boleyn, the second wife and Queen of Henry VIII. Carey and his elder sister Catherine came under the wardship of their maternal aunt Anne Boleyn, who was engaged to Henry VIII at the time. The children still had active contact with their mother, who remained on good terms with her sister, until Mary's secret elopement with a soldier, William Stafford (later Lord of Chebsey) in 1535. Henry's royal aunt was beheaded in May 1536, when he was ten years old. His mother died seven years later in 1543 on her estate in Essex.