Rowlands was born as Virginia Cathryn Rowlands in Madison, Wisconsin. Her mother, Mary Allen (Neal), was a housewife who later worked as an actress under the stage name Lady Rowlands. Her father, Edwin Myrwyn Rowlands, was a banker and state legislator. He was a member of the Wisconsin Progressive Party, and was of Welsh descent. She had a brother, David Rowlands.