Studying under Daniel Barry, he completed preparatory studies in Hopkinsville, Kentucky. He studied law under John Jay Crittenden, with whom he became lifelong friends.
When the younger Rumsey was still a child, Doctor Rumsey moved the family to Christian County, Kentucky. He moved to Greenville, Kentucky where he was admitted to the bar and practiced in Muhlenberg and surrounding counties. Rumsey was elected to the Kentucky House of Representatives in 1822.
The major questions during his tenure were those connected to the Old Court-New Court controversy.
On January 5, 1832, Rumsey married Jane Merrihew Wing. The couple had two children.
Rumsey was elected as a Whig to the Twenty-fifth Congress, serving from March 4, 1837 to March 3, 1839. The resolution unanimously passed the House, but failed in the Senate.
In spring 1838, both Rumsey"s young children contracted scarlet fever and died.
Overwhelmed with sorrow, he retired from public life at the end of his congressional term. After leaving Congress, he again resumed the practice of his profession. He died in Greenville, Kentucky in 1868 and was buried in the Old Caney Station Cemetery, near Greenville, Kentucky.
According to historian Otto Rothert, the town of Rumsey in McLean County, Kentucky was to be named after Edward Rumsey.
Out of modesty, Rumsey declined, after which the citizens agreed to a compromise whereby the town would be called Rumsey and officially be named in honor of James Rumsey.