When the Cleveland Browns drafted Brown in late 1956, the team considered him something of a consolation prize, since they had been seeking Purdue University quarterback Len Dawson. Brown quickly dispelled any regrets, as he won NFL Rookie of the Year honors in 1957. In all but two of his nine seasons, Brown rushed for over 1,000 yards, and set the standard for durability by never missing a game. In 1963, Brown carried for an unheard-of 6.4 yards per carry, over 291 carries.

He departed as the NFL record holder for both single-season (1,863 in 1963) and career rushing (12,312 yards), as well as the all-time leader in rushing touchdowns (106), total touchdowns (126), and all-purpose yards (15,549). He still holds the career record for yards per carry by a running back (5.2), and total seasons leading the NFL in all-purpose yards (5: 1958-1961, 1964), and is the only rusher in NFL history to average over 100 yards per game for a career. Brown was also a superb receiver out of the backfield, catching 262 passes for 2,499 yards and 20 touchdowns.