From 1796 Fischer was trained by Maximilian von Verschaffelt before he moved to Vienna in 1799 to study architecture under Ferdinand von Hohenberg. His first work designed in the age of only 22, the Prinz-Carl-Palais in Munich, made him famous abruptly and he became a professor for architecture at the Academy of Fine Arts, Munich in 1808. From 1811 onwards Fischer constructed the National Theatre.
He also created the draft for the extension of Munich, especially for the Brienner Strasse with the Karolinenplatz and the Königsplatz.
He died in Munich and is buried in the Alter Südfriedhof.
His drafts had enormous influence on the architecture of neo-classicism in South Germany. Fischer, who was a representant of pure classicism and rejected romantic historism, was soon displaced by Leo von Klenze as chief architect for the Bavarian court.