Homemade Sagarin Ratings

Since the early 1970s, Jeff Sagarin has been publishing sports team ratings. For most sports, including the NFL, his ratings are calculated so that the difference between two opponent's ratings, plus a home field adjustment, forecast a game's point spread. His ratings are widely recognized as some of the best around. They can be found every week of the NFL season on the USA Today site. Sagarin has never published his exact algorithms, but we can easily build a very good facsimile.

Excel has a powerful tool called "Solver." It's one of those thousand or so features that Microsoft packs into its Office products that no one ever knows about. In fact, you do don't even see it on the Tools menu until you enable it from the "Tools|Add Ins..." command.

If you go to Microsoft's on-line help site for Solver, the example problem provided is an exercise estimating point spreads for NFL games. The sample spreadsheet is for all the game scores from the 2002 season.

Basically all you do is create a table of ratings for each team. The ratings don't have to mean anything yet. For now they can be your best guess, or all ones, or anything. Solver will calculate them later. Then for each game, you calculate what the ratings suggest should be the point spread. The ratings are intended to work just like Jeff Sagarin's ratings. If team A's rating is 5 and team B's rating is 8, then when team A plays team B the point spread should be 3 in favor of team B. Factoring in a league-wide value for home field advantage, say 3, and the spread becomes 6 if team B is at home.

Next, using the LOOKUP function to grab the ratings from the table, you calculate the error between the expected spread and the actual result for each game. Square the error (as every good statistician would). In a cell, sum all the squared errors for all the games in the season. In another cell, enter a point value for home field advantage--3 points is a good initial guess. Solver takes over from here.


In the Solver dialog box, you tell it to minimize the value in the cell for the sum of squared errors. Then you tell it to do so by varying the values in the table for the team ratings and the cell for the home field advantage. (You can also add in a constraint that says the average for all the teams' ratings should be zero, so that good teams will have positive ratings and poor teams will have negative ratings.)

Solver will compute the team ratings necessary to best fit the actual point spreads. And now you have your very own homemade Sagarin ratings.

I noticed that Sagarin's average rating is 20 instead of 0, which makes sense because the average NFL score is about 20 points. So I altered the Solver constraint accordingly. For the 2007 season, including the playoffs, the homemade ratings were nearly identical to Sagarin's.

Comparison of Sagarin and MS Solver Team Ratings for 2007

Team	Sagarin	MS Solver
NE	36.4	37.7
IND	30.2	30.6
SD	30.0	28.8
GB	29.4	28.9
NYG	28.3	25.2
DAL	28.0	28.2
JAX	27.9	26.1
PHI	24.9	24.8
PIT	23.6	24.6
MIN	22.8	23.6
WAS	22.2	22.7
SEA	22.0	22.0
TEN	21.8	20.2
CHI	20.5	21.2
HOU	20.3	19.7
TB	19.8	20.5
CLE	19.4	18.8
DEN	17.5	15.8
DET	17.2	16.7
BUF	16.4	20.0
CIN	16.2	17.6
NO	16.1	17.4
ARI	16.0	16.2
OAK	15.7	14.3
NYJ	15.5	16.1
KC	15.2	15.1
CAR	15.0	14.3
MIA	13.2	11.7
BAL	13.0	14.6
ATL	10.2	9.9
SF	8.3	8.9
STL	7.1	7.8

The differences beween Sagarin's and our homemade ratings may come from the method of solving. Solver uses a "brute force" numerical iteration method, and Sagarin's method is unknown. Sagarin may also weight recent games heavier. Notice how the difference in the Giants' rating is one of the more significant. The Giants finished the 2007 season on quite a win streak.

Doug Drinen of Pro-Football-Reference.com discusses a very similar method for ranking teams based on margin of victory which he calls the Simple Rating System (SRS). His post includes a good discussion on the advantages and disadvantages of a pure margin of victory ranking system.

Sagarin actually uses two different systems. One is called Pure Points, which is based solely on point differential. This is the system which the method discussed above mimics. His other method is called Elo Chess, which considers only wins and losses, and ignores points. This system is based on a method devised to rate chess players by Arpad Elo, a physics professor and master chess player. In the next post I'll demonstrate how to mimic the Elo ratings.

The Ellsberg Paradox and 4th Down

The Romer paper and other research provide fairly conclusive evidence that NFL coaches should go for it on 4th down more often than they currently do. The Ellsberg Paradox might help explain why.

Say there are two jars of 100 balls of which some are red and some are blue. Jar A has 50 red balls and 50 blue balls. Jar B has a random unknown mix of red and blue balls. You'll be given $100 if you pick a red ball from a jar. Which jar would you choose to pick from?

In clinical experiments, people almost universally choose jar A. This is the Ellsberg Paradox, a violation of the utility theory in economics. The expected value of each choice is equal. There is a 50/50 chance of winning $100 from either jar, so we wouldn't expect one option to be significantly preferable to the other.

The Ellsberg Paradox demonstrates the difference between risk and uncertainty. Risk is measurable but uncertainty is not. People almost always prefer a known risk to an unknown uncertainty, even if the expected results are equal.

People prefer Jar A according to the equation above. U() is the utility function.

Punting seems a lot like Jar A, for which the risks and potential outcomes are known. Going for the first down seems more like Jar B, for which the potential outcomes are vague and hard to measure. So at the equilibrium point between going for it and punting, where each decision provides equal chances of ultimately winning, coaches would be heavily biased toward punting. Even beyond the equilibrium point, where going for it would be favorable, coaches would still be biased toward the relatively certain (but less favorable) outcome of the standard 40-net-yard punt.

In a strict analogy, the $100 would be a win, and the red balls would represent the probability of winning the game. There would actually be some uncertainty in each strategy, but far more uncertainty in the go-for-it strategy--perhaps something like 40 to 60 red balls in the punt jar and 20 to 80 balls in the go-for-it jar. The Ellsberg Paradox suggests coaches would naturally prefer punting, the less uncertain option. Only when the advantage of going for it is beyond obvious would a coach choose to go for the 1st down--say 10 to 20 red balls for punting and 15 to 60 red balls for going for it.

I think NFL coaches typically employ the maximin strategy. In game theory the maximin strategy is one that selects the alternative with the best worst-case-scenario. It maximizes the minimum possible payoff. This is a conservative strategy in comparison to the maximax strategy, which selects the alternative with the greatest maximum payoff.

Continuing the jar and red ball analogy, compare jar X with 10 to 90 red balls and jar Y with 30-40 red balls. Utility theory would suggest the rational option is jar X with a higher overall chance of success. The maximin choice however, would be jar Y because it has a higher minimum chance of success.

Calculating the probability distributions of a football game's outcome given the combinations of score, time remaining, field position, etc. is far more complex than being told how many red balls are in a jar. It would be overwhelming for a human brain even to attempt it. In such a situation, coaches, like everyone else, use heuristic shortcuts such as the maximin strategy. Punting on every 4th down is a known risk, especially because coaches can count on opposing coaches to follow the same strategy (which suggests that always punting is a Nash Equilibrium). Punting usually presents the best worst-case-scenario despite being a sub-optimum decision.

How Draft Experts Are Graded

It seems that everyone has a mock draft board these days. The object of this parlor game seems to be to predict which players will be picked by each team. So draft gurus tend to be judged by how many correct predictions they make. When you think about it, it's pretty ridiculous. A draft expert has to get four things right. He needs to not only evaluate players and team needs, but also evaluate all 32 teams' own perceptions of each player and its needs. And if one prediction at the top of the first round is off the mark, the house of cards collapses.

I think a much better way to evaluate draft experts is to wait several years, then see which players actually turned out to be more productive. Look at how they rated each player, not how well they read the minds of the league's GMs.

Besides, in my eyes the real value of a draft expert is be able to tell me in 30 seconds everything there is to know about that strong safety from Alcorn State my favorite team just picked up midway through the 4th round...without any notes or any hesitation...and has a full head of hair. And for that, there is only one man.

Drafting Wide Receivers

If Lions GM Matt Millen has contributed anything to the world of football player evaluation, it's that wasting a top pick on a wide receiver is very dumb idea. The fragile-egoed temperamental prima donnas that they are, WRs are known as a notoriously undependable lot. Worse, their productivity seems completely unpredictable. But it's not true.

Using data from the draft database at Pro-Football-Reference.com, all WR draft picks from 1980-2000 were evaluated by Pro Bowl selection, receiving yards per year, and years as a starter.

Pro Bowls

Receivers were classified as being selected one or more, two or more, or three or more Pro Bowls. The likelihood of each by round and by position draft order is illustrated in the two graphs below.



After the 4th WR taken in the average draft class, it becomes drastically less likely a team will find a star player. It looks like that the drop-off usually happens around halfway through the 2nd round. By the 5th WR taken and the 3rd round, WRs appear to be about equally as likely as much later picks of becoming an all-pro at about a 5-10% probability.

Receiving Yards Per Year

I normally prefer rate stats when comparing players, but comparing the records of receivers is an exception. Yards per reception is a helpful stat, but receivers create their own receptions by getting open, so yards per reception would not capture a very important part of playing WR. Yards per "target" might be better, but targets are very subjective and unstandardized statistic. For this comparison, I'll use career yards per year broken out by round and by position draft order.

We see a similar pattern as with Pro Bowl selection. After the 4th WR taken, prospects for finding a solid contributor drop significantly.

Scout Accuracy

How well to NFL team scouts and player evaluators predict the performance of WR draft picks? The table below lists the likelihood that a WR will end up having more Yds/Yr than the next WR taken in the draft. For example, the 1st WR taken ends up as better than the 2nd WR taken only 38% of the time.

WR Pick	Pr(Better)
1	0.38
2	0.67
3	0.48
4	0.81
5	0.33
6	0.62
7	0.62
8	0.38
9	0.52
10	0.52
11	0.57
12	0.57
13	0.48
14	0.43
15	0.52

It seems to be very difficult for teams to identify the better player between two closely matched potential draft picks, even for the #1 WR taken. But overall, they seem to have selected the better receivers in the top round. These results suggest a good strategy for picking a good WR with a very high draft pick is to trade down to pick the second or third WR available in the draft. Teams should have very little confidence that the highest rated WR will turn out any better than the second. On the other hand the #1 WRs tend to become the biggest stars, being selected to three or more Pro Bowls in one out of three cases.

Years as Starter

Another way to evaluate draft picks is to look at how many years they are starters for their teams. The tables below list the averages for each round and draft order.

Round	Yrs as Starter
1	5.7
2	3.3
3	2.1
4	1.3
5	0.7
6	0.5
7	0.6

WR Pick	Yrs as Starter
1	5.9
2	6.6
3	4.9
4	4.7
5	2.0
6	3.3
7	2.7
8	1.8
9	2.5
10	1.9
11	1.7
12	1.0
13	1.3
14	0.9
15	0.8

Conclusion

As temperamental as they may be, top WR draft picks really do turn out to be stars far more often than later picks. They seem to be a lot like QBs. There is a real scarcity of talent at both positions, and it is difficult to predict with much certainty which ones will pan out. A team's chances of finding a highly productive player are still better with a top pick.

Drafting Defensive Ends

Where can NFL teams find quality defensive ends in the draft? This post looks at DEs drafted in the first seven round from 1980-2000. Data comes from the great draft database at Pro-Football-Reference.com. Picks from each round and at each draft order are compared by Pro Bowl selection, sacks per year, and number of years as a primary starter.

Although sacks are obviously not the only contribution a DE makes, and some DEs benefit from certain defensive schemes more than others, I believe that it is still a valid measure. First, sacks are only used as one measure. Overall DE performance can be reflected in Pro Bowl selections and years as primary starter. Additionally, sack rate isn't used to rank individual players, just the aggregate from each round and at each spot in the draft order. Other DE contributions to the pass rush such as QB hits or hurries (or even soaking up a double team) are very difficult to quantify. However, sacks can be used as a proxy measure for the overall contribution to the pass rush. In other words, a player with a lot of sacks very probably also has a lot of hits and hurries. Across so many seasons and with so many players, the effects of scheme and other considerations largely wash out.

Unlike QBs and RBs, players who did not make a roster or make many game appearances did not pose a problem when scoring their contribution. If they didn't have any sacks, they simply got a zero.

Pro Bowl Selection

The graphs below show the likelihood that a drafted DE will be selected to the Pro Bowl. The first graph is broken out by draft round. There is a steady decline from the top rounds to the last.

The second graph is broken out by draft order, that is, which DE was the player taken as in his draft class. This graph is much more erratic.


To me, the erratic nature of the graph suggests that compared to QBs and RBs it is easier for later round picks to shine. My guess is that later pick DEs get far more opportunities than later pick QBs or RBs, partly because a team needs two DEs on the field. Plus, they're more likely to be platooned.

Sacks Per Year

Despite its limitations as a comprehensive measure of DE performance, there is a clear relationship between draft round and order and sacks. First round picks significantly outperform subsequent picks in particular.

Notice the similarity in shape from the sacks graph to the Pro Bowl graph. This suggests that DEs are being picked for the Pro Bowl on the basis of their sacks. No surprise there.

Scout Accuracy

How certain can scouts and GMs be that they're selecting the better DE compared to the next guy down the draft board? The tables below list the likelihood that the higher player selected will turn out to have a better sack rate than the next DE taken in the same draft.

DE Pick	Pr(Better)
1	0.48
2	0.67
3	0.38
4	0.71
5	0.62
6	0.33
7	0.71
8	0.57
9	0.43
10	0.67
11	0.50
12	0.47
13	0.32
14	0.58
15	0.50

Years as Primary Starter

Each DE's total years as one of their team's primary starter is a good indication of a draft pick's worth. The tables below break out years as primary starter by round and draft order.

Rnd	Yrs as Primary Starter
1	5.6
2	3.2
3	2.7
4	2.4
5	1.2
6	1.2
7	0.6

DE Pick	Yrs as Primary Starter
1	6.8
2	6.5
3	4.0
4	4.3
5	4.1
6	2.6
7	3.6
8	1.8
9	2.2
10	3.0
11	2.7
12	1.6
13	1.2
14	2.4
15	0.6

Conclusion

It appears that, on average, each draft provides two DEs in the first round who have a high probability of being an above-average player. But compared to QBs and RBs, DEs taken in later rounds may have a better opportunity to excel.

Drafting RBs

Can solid running backs really be found anywhere in the NFL draft? Years ago the conventional wisdom seemed to be that a team needed a superstar RB from the first round to win consistently. Now it seems that the conventional wisdom is that teams still need a star ball carrier, but one can be found deep in the draft. So which is it?

Data

The data consists of RB draft picks from the 1980 through 2000 drafts found at Pro-Football-Reference.com. Running back career performance was judged three ways. First, I averaged the likelihood a RB would be selected to one or more Pro Bowls by round and draft order. Second, I averaged career Yards Per Carry (YPC) by draft round and by draft order. (I also tried various ways of including receiving yards, but the variance in Yards Per Reception is very large and it distorted the data, particularly for players with relatively few receptions. Ultimately, simple YPC worked best and aligned closest with how most people see RBs. That is, Steve Sowell and Dave Meggit aren't ranked above Barry Sanders, Thurmond Thomas, or Emmit Smith.) And lastly, I averaged the number of years as the primary starter by round and by draft order.

Pro Bowl Selection

Although Pro Bowl selection is a flawed measure of career performance in many ways, it can indicate that a draft pick has "panned-out." If you sort the data by PB selection, it very quickly separates the generally productive RBs from the "three yards and a cloud of dust" guys. After looking at PBs for a number of positions now, it seems that 2 or more PB selections is a particularly good measure of career productivity, especially when judging top draft choices.

The graphs below illustrate the likelihood that a RB will be selected to one or more, two or more, and three or more PBs. The first graph is by draft round, and the second graph is by RB draft order (i.e. 1st RB taken, 2nd RB taken, etc.)




I wouldn't read too much into the spike at the 5th RB taken. It's likely just a statistical quirk, but it might be one reason why many experts believe that later round RBs are as good as early round picks.

[Edit: Some have asked why I brushed off the spike of Pro Bowls at the 5th RB taken as a quirk. If we analyze enough draft picks for various positions, as I'm in the process of doing, we're bound to see a significant bunching like this by chance once or twice. The graph is relatively continuous except in one place, where there is a depressed result at the 4th and 6th pick and the spike at the 5th. What is likely at work is that positive results in the 5th pick "bin" have randomly "stolen" positive results from the 4th and 6th bin. There were 20 RBs taken as the 5th RB in the data set, so it would only take 2 or 3 RBs who would otherwise have been the 4th or 6th pick to be bunched into the 5th pick to give us this result. Unless we had a reason to believe there is some special quality about the 5th RB taken before seeing the results, we should not interpret the data to say there is something magical about being the 5th RB taken.]

Yards Per Carry

There is probably no simpler and truer measure of running back performance than yards per carry. Of course, YPC does not belong to the RB alone. For any one RB's season, offensive line ability has a tremendous influence on his stats. But over 490 careers and over 24 years of data, the abilities of offensive lines will average itself out to a great degree, leaving career YPC a reliable estimate of true RB performance when grouped by round or draft order.

As with QBs, the biggest question is how to score draft picks with no carries or very few carries. RBs with fewer than 200 career carries tended to have extreme YPC stats. I assigned them the YPC of the 5th percentile qualifying RB, which was 3.58.

The two graphs below break out career YPC by draft round and by draft order.

The first round RBs, particularly the first couple taken, tend to significantly outperform later picks. By the 3rd or 4th round and the 7th RB taken, teams are likely getting sub-replacement level special teams fodder.

Also notice the nearly 1:1 relationship between career YPC and PB selections, including the spike at the 5th RB taken. This suggests that PB selection is merit-based and is a reasonable proxy for grading career performance.

Scouting Accuracy

How often do the scouts get it right? In other words, how often does the higher pick turn out to be better than the next pick? The two table below lists the likelihood that the higher pick will have a better career YPC than the next RB taken in the same draft. We shouldn't expect the scouts to be perfect, but this table tells us how difficult it is to predict the better player.

RB Pick	Pr(Better)
1	0.62
2	0.67
3	0.57
4	0.48
5	0.55
6	0.33
7	0.43
8	0.33
9	0.33
10	0.38

Years as Primary Starter

Another way of judging the value of draft picks is the number of years the pick has served has his team's primary starter. Top RB picks, especially the first two each year, tend to serve much longer as a primary starter.

Round	Yrs as Primary Starter
1	4.1
2	2.7
3	1.5
4	0.7
5	0.8
6	0.6
7	0.5

RB Pick	Yrs as Primary Starter
1	5.0
2	5.1
3	3.2
4	2.4
5	3.7
6	1.6
7	1.9
8	1.4
9	2.7
10	1.0
11-22	0.8

Conclusion

Top picks solidly outperform subsequent picks. The top two RBs taken tend to almost be in a class to themselves, then there is a steady decline in expected performance until the 8th RB taken, at which point there is very little to be expected from a pick.

So do teams need a superstar #1 pick RB to win, or can they find a premier runner deep in the draft? Which conventional wisdom was right? My theory is it's neither.

I think most people still grade RBs in terms of total yards, whether it's for a single game or for a season. Even though it should be well known now that winning leads to running, rather than vice versa, commentators and analysts continue to count 100 yard games or 1000 yard seasons as measures of RB effectiveness.

But even below-average RBs on winning teams with good passing games and good defenses will tend to accumulate large chunks of total yards due to frequent carries. Even a RB who was a 5th round pick on a great passing team will appear much better than he truly is. I believe that might explain the perception that solid RBs can be found anywhere in the draft.

The better RBs really do come from the top picks. It's just that they're not that important, or at least they're not as important as they were in the 1970s before the NFL became a passing league. Plus, our understanding of which RBs are truly the very good ones is distorted by analysts who insist on total yards as the best measure of RB performance.