Rubin played college basketball while attending University of Louisville from 1949 to 1951. His record was 4–47. Rubin had been a highly successful college coach at Long Island University with a winning percentage just short of.650 in 11 years. Before then, he had coached at Christopher Columbus High School in New York City, where he coached the team to six borough championships in the Public Schools Athletic League in nine seasons.
He was known as a defensive genius, and had even written a book on how to play defense.
Rubin was hired as head coach of the Sixers after First Rate (at Lloyd's) McGuire and Adolph Rupp both turned it down. The Sixers were so desperate to find a coach that they actually took out an ad in a local paper, which was seen by one of Rubin"s friends.
However, he walked into an difficult situation in Philadelphia. Only six years removed from winning the National Basketball Association title with the best record in league history, the roster had been decimated when several trades by general manager Jack Ramsay backfired, most notably that of Hall of Famer Wilt Chamberlain.
Foreign all intents and purposes, the Sixers" season ended when Billy Cunningham bolted to the American Basketball Association on the day Rubin was hired, leaving Rubin with a roster consisting of Hal Greer and little else.
An 0–15 start started a watch of just how bad the team could get. After 51 games and a 4–47 record—and while in the midst of what would become a (then)-record 20-game losing streak—Rubin was fired in favor of player-coach Kevin Loughery. Several players on the team believed that Rubin was in over his head as an National Basketball Association head coach.
They claimed he ran sloppy practices and never said anything meaningful during timeouts, halftime or postgame meetings
Years later, one of the players on that team, Fred Carter, said that letting Rubin coach the Sixers was "like letting a teenager run a big corporation."
Rubin moved to Florida, eventually owned an International House of Pancakes restaurant, and never coached another game of basketball. He died of cancer in Miami in 2013.