James studied the classics at Slate Ridge Presbyterian Church school and at an academy in Pequea, Pennsylvania.
At the age of eighteen Ross was induced by the Rev. John McMillan, a close friend of the Ross family, to go to western Pennsylvania, where he taught Latin and Greek in McMillan's academy near Canonsburg (now Washington and Jefferson College, Washington, Pa. ). He had originally intended to enter the ministry, but while he was at Canonsburg, Hugh Henry Brackenridge, a Pittsburgh lawyer, persuaded him to take up law, and in 1782 encouraged him to continue his studies at Philadelphia. He returned to Washington County in 1784 and was admitted to the bar.
Specializing in land cases, he soon acquired a large practice, and in 1795 moved to Pittsburgh. He was attorney for President Washington's estates in western Pennsylvania and numbered among his clients prominent and wealthy business men. His first connection with state politics was as a member of the Pennsylvania constitutional convention in 1789-90. A member of the committee that drafted the new frame of government, he was a stanch Federalist and attracted attention by his zealous advocacy of a clause for religious liberty similar to that in the federal Constitution.
During the Whiskey Insurrection (1794) he used his influence to restrain popular fury against the federal government. President Washington appointed him one of the federal commissioners to treat with the insurgents and his efforts were in a large measure responsible for the amicable settlement of the uprising, and the saving his friend Brackenridge from prosecution for treason.
In 1794 the Pennsylvania legislature elected him to the United States Senate in place of Albert Gallatin, who was disqualified on account of the residence requirement. Reelected in 1797, he served until 1803, and in 1799 was president pro tempore of the Senate. A firm believer in Hamiltonian ideas and policies, he worked diligently, though unsuccessfully, in 1800, to keep Pennsylvania in the Federalist ranks and to ensure a national victory for his party by urging the passage of an act under which the legality of electoral votes for president and vice-president would have been decided by a grand committee composed of the chief justice and six members from each house of Congress.
Under the Republican administration he defended Federalist legislation against Jeffersonian attacks, notably the excise law and the Judiciary Act of 1801, asserting that the repeal of the latter would erect Congress into "a complete tyranny" and render the judiciary totally subservient to Congress (Annals of Congress, 7 Cong. , 1 Sess. , p. 166).
A series of resolutions introduced by him on February 16, 1803, following Spain's withdrawal of the right of deposit at New Orleans and designed to embarrass the administration and provoke war with Spain, demanded the immediate seizure of the mouth of the Mississippi River, the fortification of its banks, and then negotiations for navigation advantages. In 1799, 1802, and 1808 he was Federalist candidate for governor of Pennsylvania, but the high tide of Jeffersonianism, his liberal religious views, and his refusal to canvass the state conspired in each instance to defeat him.
From 1816 to 1833 he was president of the Pittsburgh select council, but otherwise he was not active in politics after 1808, his law practice and land speculations, which proved highly profitable, engaging his attention. He died at Allegheny City, now a part of Pittsburgh.
His wife, whom Ross married on January 13, 1791, was Ann, daughter of George Woods, of Bedford, Pennsylvania.