In the war of Heiji, he led troops of 1,000 men in a battle with Yoshitomo Minamoto near the Ikuhomon Gate of the Palace. Routed by Yoshitomo, he fled to Rokuhara. He afterward besieged the Ninnaji Temple together with Norimori and captured Nobuyori Fujiwara. For the exploit he was given the title of Lord of Owari and later was promoted to Gon-dainagon (Vice-Minister) (1183). He defied the order to fight Minamoto warriors who had risen under Yoritomo's leadership after 1180.
When Munemori led the Taira family in its retreat to the west from Kyoto (1183), Yorimori stayed in Kyoto and served ex-Emperor Goshirakawa, counting on the good will of Yoritomo, because is mother had saved Yoritomo from death when he was a boy. Then he went to Kamakura at the invitation of Yoritomo (1183) was well treated and recovered his territory through Yoritomo's recommendation to the Court.
He became a Buddhist priest (1185) and was popularly known as Ikedono.