Rinne was developed in Kärpät playing predominantly for their junior team between 2000–2003 and after that, for their men's team in championship-winning years 2004 and 2005, mostly as a backup of Niklas Bäckström.
Rinne was drafted 258th overall in the 8th round of the 2004 NHL Entry Draft by the Nashville Predators of the National Hockey League (NHL). He was signed by the Predators on August 23, 2005 and began his North American playing career with the Predators' top minor league affiliate, the Milwaukee Admirals of the American Hockey League (AHL). Initially Rinne was slated to be the backup for Brian Finley, but Rinne became the starter as a consequence of Finley's poor play. In the summer of 2006, Rinne was "roughed up by a couple of unknown assailants" and during the fight he dislocated his shoulder. As a result, he had to have surgery and missed the first four months of the season.
Rinne appeared in two NHL games with Nashville during the 2005–06 season. He compiled a record of 1–1–0 with a .900 save percentage and a GAA of 3.80.
Following Dan Ellis's stellar performance in the 2007-2008 Nashville Predators campaign, starter Chris Mason was traded to the St. Louis Blues and Rinne was called up to serve as back-up to Ellis for the 2008-09 season.
On December 1, 2008, Rinne became only the second Predators rookie goaltender ever to record a shutout. In February 2009, Rinne recorded nine wins in twelve games and had a 0.944 save percentage and a 1.72 goals against average. He was selected as the NHL rookie of the month. On March 14, Rinne set a Predators franchise record with his seventh shutout of the season. Rinne was one of two goaltenders to play for Finland in the 2009 IIHF World Championship of hockey, the other one being Karri Rämö. Rinne started the tournament with a shutout in his first game in the Finnish national hockey team, saving 30 shots.
In the 2010-11 season opener, Rinne allowed a goal before colliding with Ducks forward Troy Bodie and suffering a lower-body injury. Anders Lindback relieved him and went on to start three games before Rinne returned, only to go down again in December. Since returning on December 23, Rinne remained healthy and put up some of the better numbers of his career, including a 1.71 GAA and .946 save percentage in the month of January. At the conclusion of the season came many accolades, Rinne was named one of the three finalists for the Vezina Trophy, conferred to the best goaltender in the NHL as judged by the league's General Managers. He finished 4th in the Hart Trophy voting and was also named an NHL Second Team All-Star. Rinne led the Predators to their first ever playoff series win in franchise history, in the 2010–11 playoffs.
After shutting out the Anaheim Ducks on October 29, 2011, Rinne set the mark for the Predators franchise record for shutouts with 22.