Milner was educated at Wellington College and at Trinity College, Cambridge.
Having captained Wellington as a left-handed opening batsman and a left-arm medium-pace bowler, he appeared as a lower-order batsman and bowler in his two Cambridge matches. He took five wickets and scored 20 and 4 in the first game against an invitational amateur side, but was not successful in the second, and did not appear again. Milner had a varied career after leaving Cambridge.
According to his own short obituary in The Times, Milner acted as "controller" to successive generations of the Stanley family, the Earls of Derby.
He also served briefly as a civil servant: he was assistant private secretary to the Under-Secretary for War in 1902 and private secretary from 1905 to 1906.