He started as a referee in 1979 and was a referee in Fédération internationale de football association from 1992 to 2002. Olsen officiated in 2002 World Cup qualifying, refereeing the match between Spain and Bosnia. He has refereed over 30 national matches at different age groups, both men and women, in addition to 28 matches in the European Cup.
In 1991, he refereed the Norwegian Cup Final between Strømsgodset and Rosenborg, as well as the Norwegian Cup Final between Vålerenga and Strømsgodset in 1997.
He also refereed the European Championship for U-18 in England in 1993.