Brown was the first black player in the history of his first team, Stoke City. Brown also played for Watford, making a total of 212 appearances in the Football League for both clubs. He joined Stoke City in 1939 and played for the club during World World War II whilst also serving with the army.
He was used a back-up utility player by manager Bob McGrory eventually playing regularly in 1950-1951 where he played in 29 matches.
He scored eight goals in 16 matches during the 1952-1953 campaign the second highest as Stoke suffered relegation from the First Division. Brown now seeking regular first team football left the Victoria Ground in the summer of 1953 for Third Division South side Watford.
He spent the first season with the "Hornets" as a centre forward and he scored 21 goals as Watford finished in 4th position. He remained at Vicarage Road for four more seasons before ending his career with non-league Chelmsford City.