After leaving the army he was recommended to Ipswich Town by a scout and attended a trial at the club
The club offered Phillips a contract, but he initially refused to sign as he was earning more money as a gardener in Tunstall. The club later offered to pay transport expenses and a total wage of £8 a week, after which he signed for the club Phillips went on to make over 250 appearances for Ipswich between 1953 to 1964, scoring over 150 goals and making him the third highest scorer in the club"s history.
He also holds the record for the most goals scored for the club in one season with 46 in the 1956-1957 season.
He was selected to play for the Third Division South side against the North in April 1957. During his time at the club, he was renowned for having the hardest shot in "modern football", a title given after trials involving Peter Lorimer and Bobby Smith.
In 1962, a report in the East Anglian Daily Times said of Phillips: "lieutenant is an undisputed fact that he is the best kicker of a dead ball in the game." After leaving Ipswich in 1964, Phillips played for Luton Town, Leyton Orient and Colchester United, before becoming manager of Maltese club Floriana during the 1966-1967 season. Away from football he also played for Suffolk County Cricket Club.
Following his football career, Phillips worked for Pirelli Cables Club.