NFL Stats Overview

This post is intended to summarize the statistical methods commonly used throughout this site. The statistical model uses each team's efficiency statistics in running and passing, on both offense and defense, to predict team wins. Turnover rates and penalties are also included. Multivariate linear regression was used to estimate total season wins, and multivariate logit (non-linear logistic) regression was used to estimate individual game probabilities.

Efficiency stats are defined by yards per play. Passing efficiency is defined as the average yards gained (or lost) per passing play. I included yards lost in sacks in both offensive and defensive pass efficiency. Likewise, run efficiency is defined as yards per run. Turnovers are also defined as efficiencies. Interception efficiency is defined as interceptions per pass play. Fumbles are defined as fumbles per any play--both run and pass plays can result in fumbles.

Efficiency stats were found to be correlated with wins much better than total yardage stats. Teams with poor defenses that find themselves trailing (and likely to lose) tend to accumulate large amounts of gross passing yards, but only because of frequent attempts. Teams that are ahead (and likely to win) tend to accumulate large amounts of gross running yards. In both cases it's often the winning or losing that lead to the yardage, not the other way around. For this reason, efficiency stats are almost always a better measure of a team's proficiency than gross yardage stats.

Once I established a workable, logical, predictive, and statistically significant model of winning and losing, there were any number of interesting applications. I could now predict individual games. Summing the total probabilities of all 256 games in the season estimates the probabilities of a total number of wins for each team. Toward the end of the season, the game-by-game model can also be used to compare likelihoods of selected teams earning a playoff spot, or will capture a wildcard berth. I could also create very accurate power (or efficiency) rankings. Another application was that I could estimate how lucky each team was, and how much luck played a part in determining game and season outcomes.

I'm not just trying to predict games. I'm trying to understand the game itself. For example, which is more important--running or passing? Offense or defense? Interceptions or fumbles? Do special teams matter? But answering those questions accurately depends on how accurate the model is, which can only be judged by how well it predicts wins. We each may have intuitive answers to these questions, but statistics is one of the best ways to test our hunches. We can even quantify them.

2006 was the first season in which I did all of this number crunching. I needed some minimum amount of data to use for the efficiency stats, so I started predictions after week 4. As the season went on, I had a larger data set. Ulitmately, here is how my predictions fared:

At first glance, 63% correct may not appear too impressive, especially for all that work I did. But almost 2 out of 3 games correct isn't too bad, especially when compared to the national consensus. Each week, I recorded not only my own results, but those of the Las Vegas odds makers as well. (Although my purposes here are not related to gambling, I'll use the betting lines as a benchmark.) The Vegas line was only accurate in picking winners 58.2% of the time for the entire '06 season. Vegas' record was slightly worse from week 5 onward, at 57.1%.

By end of the 2006 regular season, my model was correct significantly more often than the Vegas line, by 63% to 57%. On the surface, it may not sound as impressive as it really is. But think of game prediction this way: A monkey will guess winners correctly 50% of the time. So the real question is: how much better than 50% is the model? In this case, the efficiency model added almost twice the predictive power as the Vegas line (the consensus favorite).

The 2007 season was more predictable. The model was 70.8% correct compared to just 66.7% for the Vegas consensus. Over the past two years it has been the most accurate prediction system published.

Also consider that no model could be 100% correct. Upsets happen for many reasons. Some games are very evenly matched, so the favorite has very little advantage to begin with. Additionally, luck plays a large role in determining outcomes. It's hard to say exactly how well the theoretically best possible model could do, but from my experience it seems it would be something just under an 80% correct rate.

Ultimately, we're not talking about 71% vs. 67%. We're asking how far from 50% and how close to 80% can a prediction model get.

Last Word on the Draft and Wins

Recap

Through a series of regression models, I tested the hypothesis that more and higher draft picks translated into wins on the field. It became clear that neither conventional draft points nor Massey-Thaler surplus values were significant predictors of team wins the following season when previous year wins were taken into account.

One important point: It goes without saying that what really matters is the players who are picked. The most over-cited example contrasts Tom Brady, picked in the 6th round, and Ryan Leaf, picked #2 overall. The same principle holds throughout the draft. What is truly important is how good the chosen players are, and not how high the pick was. However, on average, players picked higher definitively outperform players picked lower. Over time the significance of any one player outperforming or underperforming his place in the draft should average out. Theoretically, in a large enough sample, draft picks (not the players) should correlate with team talent levels and therefore wins too.

Are draft pick values non-significant because they don't matter, or because we don't have enough data? Consider M-T values in 2005 as a case study. The coefficient for M-T value without accounting for previous year's wins is 0.86 wins per million $ of surplus value. The coefficient drops to 0.24 when accounting for previous year's wins, but we cannot have much confidence in that number because it is not significant (p=0.42).

Example 1

In 2005, the team with the most M-T value was TEN with $5.3 million in surplus. The team with the least M-T value was NYG with $2.2 million. (The average was $3.7 million.) That's a difference of $3.1 million between the top and bottom teams.

Using the 0.24 wins/$value coefficent, we get:
3.1 x 0.24 = 0.75 wins.

Between the top team and bottom team in terms of M-T draft value, we could estimate a difference of 0.75 wins. That's a very rough estimate because of the lack of statistical significance, but it is significant in NFL terms. 0.75 games would be a lot in a 16 game season.

Example 2

Let's do the same drill with conventional draft points in the same year, 2005. Without considering the previous year's record, the coefficient for draft points was 0.0022 wins/draft pt. The coefficient drops to 0.00040 when accounting for previous year's wins, but it is not signficant (p=0.36).

In 2005, the team with the most draft points was SF with 4049 and the team with the least was BUF with 603 for a difference of 3446. The average that year was 1898 points.

Using the 0.00040 wins/pt coefficient, we get:
0.00040 x 3446 = 1.37 wins.

1.37 wins is very significant in NFL terms in a 16-game season, especially considering that the difference between 9-7 and 7-9 sometimes means going to the playoffs or firing the head coach. We just can't have much confidence that the real number is 1.37 without more data.

Conclusion

This leads me to suspect that conventional draft points, M-T values, or other measures of draft "capital" are probably important to a team's fortunes. I simply don't have a database large enough (yet) to determine a significant coefficient. But in the case studies above give us an idea of the likely potential magnitude of the effects. Admittedly, it is possible that the effects are zero or near-zero (or even negative). There is a third possibility--that draft picks do matter, but neither conventional draft points nor M-T surplus are good measures of their value. Another system of values, one that is calibrated to the effect the picks have on wins, might prove to be the best method of valuing draft picks.

The effect of the previous year's wins is powerful. The constant churn of poor teams improving and good teams getting worse dominates the effect of either draft variable. Ignoring draft variables, teams with 3 wins tend to improve to 7 wins the following year. Teams with 12 wins one year tend to stumble to 9 wins the next. I suspect this is partially why it is difficult to distill the true effect on wins of draft picks, assuming there is one.

The reasons for the churn may include strength of schedule, salary cap considerations, and the draft. But luck and other mathematical reasons may also be at work. I plan to address this in a later post.

Do Draft Picks Equate to Wins? 3

Does the Massey-Thaler (M-T) surplus value of picks translate into wins? I repeated the analysis I did with conventional draft points.

First, I performed a simple single-variable regression with "DeltaWins" (the change in team wins from the previous year) as the dependent variable and M-T surplus value as the independent variable. M-T surplus was significant with an r-squared of 0.03. The coefficient was 0.86, which means that the difference between the #1 team and the #32 team in the draft equates to 0.34 wins.

This is far less than the r-squared for draft surplus (0.26). But keep in mind, M-T surplus is not as directly related to previous year's record as draft points. We already know that a team's previous year's record is dominantly predictive of the following year's win total, regardless of draft points.

To account for the other factors that produce the tendency for poor teams to improve and good teams to get worse, the previous year's wins was added to the model. M-T Value is no longer significant (p=0.42) and last year's wins (LASTWINS) dominates. Adjusted r-squared = 0.38, which was the same as for conventional points.

I ran several more regressions and got very similar results to the conventional draft point models. Adding various combinations of M-T values, previous year values, and cumulative values did not improve the r-squared found by using last year's wins alone, which is about 0.37. Below is the table of results. Asterisks denote significance and negative coefficents are in red.

If draft points don't matter (compared to other factors), then M-T values matter even less.

Do Draft Picks Equate to Wins? 2

Teams that have losing records tend to improve and teams that have winning records tend to get worse. There may be several reasons including the draft, but scheduling and other effects may also be factors. This isn't any revelation, except that it severely complicates the question of whether draft picks can be converted into wins.

My first regression was a simple single-variable model that estimated net wins based on a teams conventional draft points in the immediately preceeding draft. This time, the model will include the teams' previous year's record to account for the other factors that explain the tendency for teams to regress to the mean--better teams get worse, bad teams get better.

Because draft points are mostly a function of a team's previous record, I expected a large amount of overlap in their effect on future wins. To really isolate and understand each variable's effect on wins, it would take some creative "differential" regression--we need to run several different regressions and compare the results.

Including both draft points and last year's win totals in the model reveals the following. Last year's record is highly significant and the coefficient is negative (as expected--more wins last year means less this year). Draft points, however, is not significant (p=0.36) but still postitive. The adjusted r-squared is 0.38.


Regressing last year's wins onto the following year's wins (without any draft variables) results in a virtually identical r-squared of 0.38. This means that draft points do not contribute any significant predictive power to the model. The other factors that explain why teams regress to the mean dominate the effect of the draft. This is strong evidence that draft picks don't matter, at least when compared to other explanations for improvements in team wins.

I conducted numerous other regressions. For example, I used draft points from 1 and 2 years preceeding the current year, theorizing that draft picks require a year or two to mature before impacting a team's record. I also used the cumulative draft points from the preceeding years. Curiously, draft points from 1 year ago was significant at the p=0.10 level, but its coefficient was negative. In other words, teams with high picks the year preceeding the season in question tended to get worse.

The table below summarizes many of the regression models I ran. Each row is one model. Significance is denoted by asterisks (p=0.10, 0.050, and 0.01 levels). Negative coefficients are in red.



No matter how it's sliced, previous year wins dominates the model.

Do Draft Picks Equate to Wins? 1

My theory was that yes, more and higher draft picks typically mean increased success to their team. Keep in mind I am talking about the draft picks in the abstract--not necessarily the players themselves but simply the selection positions. (Credit goes to Doug Drinen who originally attempted to use regression to test the connection between draft points and wins.)

To test the theory, I created a variable called DELTAWINS which is change in total wins from year to year for each team from 2002-06. This was my dependent variable. For my independent variables I used conventional draft pick point values. In other words, I used each team's total draft points from the immediately preceeding draft to see if it significantly correlated with an improvement in each team's season win total.

Simply put, higher draft picks--and more of them--should lead to more wins the following year. If so, the regression coefficient for conventional draft pick point values would be expected to be positive and significant. Actually, the coefficient is 0.0022 and is strongly significant, with an r-squared of 0.26. (The constant for the equation is -4.2.)

The team with the #1 pick in each round, the team with the worst record, has a total of 3957 points. It would be expected to improve by:

-4.2 + 0.0022 x 3957 = 4.6 wins.

The team with the #32 pick in each round, the Super Bowl champion, has a total of 1020 points. It would be expected to get worse by:

-4.2 + 0.0022 x 1020 = -1.9 wins.

However, the regression has problems. First, the distribution of draft points is not normal--a bell curve. (Someone who points this out is not normal either, I admit.) It is a gamma distribution. One of the prerequisites for linear regression is that the variables's distributions are approximately normal.

For now, however, we'll ignore that problem because there is a bigger one. NFL teams with losing records tend to improve and ones with winning records tend to get worse. The draft may be one reason, as my hypothesis states, but there are others such as "strength of schedule" or salary cap boom/bust cycles--more on the other possible reasons later. For now, we'll accept that the model needs to account for a teams previous W-L record.

Robust M-T Draft Team Comparisons

Previously, I ranked teams in terms of how well they converted conventional draft points into surplus performance value as calculated by the Massey-Thaler (M-T) study. Teams were ranked by draft points, then ranked by M-T value. The difference between each team's rankings indicates how well each team played the Massey-Thaler game.

Comparing ordinal rankings is not the most sophisticated analysis. Statisically speaking, continuous variables are more robust than ordinal variables for analyzing correlations.

So to see if a team, or teams, stood out at maximizing M-T surplus value, I regressed conventional draft points onto M-T value. In other words, I performed a simple regression with draft points as the independent variable and M-T value as the dependent variable. The sample includes the '02-'06 drafts.

The "residual values" of the regression are the differences between the predicted team M-T values and actual M-T values. If teams did no trading, and made no effort to increase M-T value, then the residuals should be zero, or close to zero. If a team made an effort to maximize M-T value then the residual for that team would be positive. Negative residuals indicate a team has "squandered" value in M-T terms.

Below is the table of residuals. For comparison, the table includes the difference in conventional points ranking and M-T value ranking I previously used to compare teams. Also listed are the team wins for the sample time period '02-'06.

The Titans and Rams are at the top of the list, while the Redskins again are at the bottom. There seems to be a drop-off from Tennessee and St. Louis to the rest of the pack. Perhaps they are teams that have struggled with salary cap issues during the time period, then were forced to makee draft pick trades to minimize the cap hit of their picks.

The average number of wins for the period is 40. The big question is 'how do team wins correlate with M-T value over the same time period?' The correlation coefficient is 0.44 (r-squared = 0.19). About 20% of the variance in wins can be accounted for by Massey-Thaler surplus value.

This is very remarkable. Conventional draft points should correlate very negatively with team wins. The more a team loses, the higher the picks it gets. In fact, the correlation between team wins and conventional draft points is -0.65.

M-T values are not as directly related to a team's previous record as conventional draft points. However, the losing teams with the higher picks still start with more M-T value than the winning teams with later picks. Therefore, if there is no "M-T phenomenon," then we'd expect the correlation between M-T "effort" and wins to be weaker than between conventional draft points and wins, but also decidedly negative.

Instead, we find that the correlation is solidly and significantly postive. (The significance threshold is 0.15.)

Overpaid Free Agent Positions?

In the last couple posts, the relative draft value of each position was calculated by summing the conventional draft points for each position. The same can be done with Massey-Thaler draft surplus value for each position. Here is the result:

RB, WR, TE, and CB top the list. But what does that actually mean? It's not as clear as draft point value. The point value from the conventional draft chart represents the draft "capital" spent to select a player. But the M-T surplus value represents the typical performance of the pick above his contract cost. The value of the pick's performance is based on the cost of a free agent with equivalent performance.

So what the table above actually shows is how each draft position overperforms compared to free agents at the same position. So the positions at the top of the list are the positions at which free agents are the most overpaid relative to draft picks.

Take RB for example, the position at the very top of the list. The data basically says that teams are probably better served by employing drafted RBs rather than free agent RBs. One example that comes to mind is the recent experience of the Colts. Marshall Faulk was an excellent player throughout his initial draft contract but the Colts did not resign him. They drafted Egderrin James who performed practically at the same level, and for half the price. When James' contract came up, the Colts allowed him to leave and replaced him with another draft pick--Joseph Addai, who was more than talented enough to help the team win a championship.

The same might be said for WRs, CBs, and TEs.

"Day 1" Draft Values by Position

In the last post, I calculated the value that NFL teams place on each position during the draft. The analysis accounted for the number of players required at each position in terms of players on the field (1 QB, 2 WRs, etc.) and in terms of roster composition (3 QBs, 6 WRs). I considered all 7 rounds of the draft from '99-'06. But "impact" players, those who are most likely to become starters and help win games, typically come from the first 3 rounds, rare exceptions excluded. So I performed the same analysis, but this time only considered players taken on "Day 1" of the draft.

The first table ranks each position by 1st-day draft points per player on the field. For comparison, I also list the relative ranking of each position per player on the field from all 7 rounds.

The second table ranks each position by 1st-day draft points per roster composition. Again, I also list the relative ranking per roster composition from all 7 rounds.


There is no difference in the relative rankings of the positions between day 1 and day 2. That should have been obvious to me from the outset because the value of picks drops off so sharply from the first round.