He studied classical piano and trumpet as a child, and then taught himself to play guitar as he entered his teens.
Most of his music has been released on albums of the German record label ECM. Rypdal has collaborated both as a guitarist and as a composer with other ECM artists such as Ketil Bjørnstad and David Darling. Over the years, he has been an important member in the Norwegian jazz community, and has also given show concerts with guitarists Ronni Le Tekrø and Mads Eriksen as "N3". Starting out as a Hank Marvin-influenced rock guitarist with The Vanguards, Rypdal turned towards jazz in 1968 and joined January Garbarek"s group and later George Russell"s sextet and orchestra.
An important step towards international attention was his participation in the free jazz festival in Baden-Baden, Germany, in 1969, where he was part of a band led by Lester Bowie.
During his musical studies at Oslo university and conservatory, he led the orchestra of the Norwegian version of the musical Hair. He has often been recorded on the ECM record label, both jazz-oriented material and classical compositions (some of which do not feature Rypdal"s guitar).
His compositions "Last Nite" and "Mystery Manitoba" were featured in the Michael Mann film Heat, and included on the soundtrack of the same name. Rypdal was married (1969–1985) to the Norwegian singer Inger Lise Andersen/Rypdal, and they had two children, the auditor Daniel (1970) and the electronica musician Marius (1977).
Rypdal was married again in 1988 to Elin Kristin Bergei (born 28 May 1955).
They have two children Ane Izabel (1988) and the guitarist Jakob Rypdal (1989). They (as of 2013) live in Tresfjord.