Brent was born into a prominent Catholic family, for which the Brent Society is named. His mother was Ann Carroll, whose brother John Carroll was the first Catholic Bishop appointed for the United States. In 1789 Brent married Mary Young — the daughter of Notley Young, a plantation owner in Prince George"s County, Maryland.
He soon took over his father"s businesses, selling sandstone to the United States. government for the White House, United States. Capitol, and other early construction projects in the District of Columbia and thereby becoming one of the capital"s most prominent merchants.