In November 1952, he signed for Southampton for a fee of £4,000, making his debut at inside-right in a 1–1 draw away to Fulham on 26 December 1952. Williams made a total of ten appearances (at either inside-right or inside-left) during the 1952-1953 season at the end of which Southampton were relegated to Division Three. Williams was only 5 ft 4in tall and was probably the smallest forward ever to play for Southampton, but he had a stocky build and was able to "mix it" with the toughest defenders and soon became a first-team regular.
In the 1953-1954 season, he became a fixture at inside-right until January, when he was dropped with Walker moving across from the left and Henry Horton taking over at inside-left.
Williams returned for the last four matches of the season, in which the Saints finished in sixth place, after having been first or second until late-January. In the next season, Williams only made a handful of appearances with Tommy Mulgrew and Johnny Walker now firmly established at inside-right and left respectively.
In his three seasons at The Dell, Williams scored seven goals from 41 league appearances. In July 1955, he returned to Hereford United and went on to become the second highest goalscorer in the club"s history.
In total he scored 154 goals in 357 appearances for Hereford.
In April 1961, he was rewarded with a benefit match at Edgar Street against Southampton. He later played for Worcester City and Cinderford Town. After retiring from football, Williams ran a window-cleaning business in Hereford.
He died in October 2011.