Born in Split, Varvodić began his career in the youth of hometown club Hajduk Split, making his professional debut in 2006. In 2007, he spent a loan spell at Mosor and the following two seasons in Germany with 1. Football Club Kölane After the two year loan finished, 1.
Football Club Köln signed Varvodić on a permanent deal.
He made his Bundesliga debut on 15 October 2010. On 23 June 2012, he signed a two-year deal to with club Qarabağ.
In February 2014 Varvodić moved to Greuther Fürth.