Williams began playing piano at age 13 and settled on trumpet soon afterwards. He may have played with Cow Cow Davenport while still a teenager. In the 1930s he played in the territory bands Trianon Crackerjacks and Brown Skin Models, and acted as musical director for the Dixie Rhythm Girls.
Around 1940 he left Alabama for Philadelphia, where he played with Tiny Bradshaw before joining the United States. Army during World World War World War II After the war Billy Eckstine hired Williams, and following this he worked with John Kirby and Billy Kyle.
In 1949 he began the first of several stints with Duke Ellington, who bestowed upon him the nickname "Cadillac". In 1951 he left Ellington"s employ and moved to Paris, where he led his own bands and recorded for French labels.
He returned to Ellington in 1956, and played with him again in 1969 on a tour of Europe. He died in the Netherlands in 1973.