Lovell received his early training in the South Grammar School, of which his father was master, graduated from Harvard in 1756 and took a post-graduate course the following year.
Lovell started to work as an usher in his father's school, a position which he held acceptably for eighteen years. Such was his scholarship that he delivered an oration in Latin in the chapel of Harvard College, February 19, 1760, at the funeral of Henry Flynt, a long-time tutor in the college.
Ten years later he was chosen as the first orator to commemorate the Boston Massacre. The oration which he delivered on April 2, 1771 placed him amongst the stanchest opponents of British measures respecting the colonies. The South Grammar School was closed by the British military authorities in April 1775, and following the battle of Bunker Hill James was arrested for spying and giving intelligence to the rebels and in 1776 was sent as a prisoner to Halifax. It happened that his father took up his residence in Halifax at about the same time, as a Loyalist refugee. After some delay, owing to the fact that some of Lovell's "billets, " as he called them, fell into the hands of General Howe, an exchange was effected in the autumn of 1776, and Lovell returned to Boston "to the no small joy of the inhabitants" of that city. Within a few days of his landing he was chosen as a delegate to the Continental Congress, taking his seat February 4, 1777, at Baltimore.
From the first Lovell took an active part in the proceedings of Congress, distinguishing himself at once for industry and zeal. Investigation of the conduct of Schuyler and conferences with Gates enhanced his conception of the prowess of the latter general, and partiality for Gates, which, after Saratoga, attained a degree of perfervid devotion, led him straight into the ranks of the critics of Washington. What part he may have had in the actual formation of the plot known as the Conway Cabal is not definitely known, but that he fomented it with all the power that was in him, is sufficiently evidenced by his letters to Gates and other intimate correspondents. Lovell surpassed all his colleagues in his vocabulary of sneers and sarcasm directed at Washington yet, like many another, in 1789 he could profess a pious devotion to the "demi-god"--and beseech him for an appointment to office.
Lovell's early appointment to the committee on foreign applications must have had no small share in shaping his subsequent course. Congress was besieged by a horde of French officers seeking commissions, and Lovell was one of the few members who knew French. "These Frenchmen have used me up quite, " he wrote in June 1777, and his mood had not changed, when, a month later, he met Lafayette and sought vainly to chill the ardor of that young enthusiast. But the most far-reaching consequence of this episode was the distrust, mounting to fierce hostility, engendered in a large group in Congress toward Silas Deane who was in a measure responsible for the coming of the Frenchmen. The anti-Deane party, of which Lovell was one of the most rabid, were not able to destroy Franklin, but they succeeded in hounding the life out of poor Deane.
He was appointed a member of the committee for foreign affairs on May 26, 1777. The committee was neglected by Congress, and for months at a time Lovell was all that was left of it. Members came and members went, but Lovell stayed on, never once in five years so much as visiting his wife and children. Diligent to a fault, he kept his seat all day in Congress, then spent long hours at night "quilldriving, " as he expressed it. Marbois described Lovell as "Homme de capacité, souple, insinuant laborieux, intelligent, " but little conversant with foreign affairs. Lovell, nevertheless, sought in the autumn of 1779 to be placed where he might catch Franklin's mantle when it should fall. He had already done what he could to loosen it. He was not a diplomat, though he was liberally gifted in intrigue and loved mystery and mystification. A useful member of Congress in many ways, serving on innumerable committees, sometimes taking high ground, he nevertheless vitiated his career by his intense partisanship. For reasons that do not wholly appear Lovell quitted Congress in April 1782, apparently chagrined over the failure of so many of his cherished ambitions.
He became receiver of continental taxes in Boston, then (in 1788) collector of customs for the state of Massachusetts, and on August 3, 1789, was appointed naval officer for the district of Boston and Charlestown. This office he held for the remainder of his life.
On November 24, 1760 Lovell married Mary Middleton, daughter of Alexander Middleton. His eldest son, also named James, made a creditable record as an officer in the Revolution and lived till 1850; his grandson, Joseph Lovell, rose to be surgeon-general of the United States; and his great-grandson, Mansfield Lovell, became a Confederate general.