Last Word on the Draft and Wins

Recap

Through a series of regression models, I tested the hypothesis that more and higher draft picks translated into wins on the field. It became clear that neither conventional draft points nor Massey-Thaler surplus values were significant predictors of team wins the following season when previous year wins were taken into account.

One important point: It goes without saying that what really matters is the players who are picked. The most over-cited example contrasts Tom Brady, picked in the 6th round, and Ryan Leaf, picked #2 overall. The same principle holds throughout the draft. What is truly important is how good the chosen players are, and not how high the pick was. However, on average, players picked higher definitively outperform players picked lower. Over time the significance of any one player outperforming or underperforming his place in the draft should average out. Theoretically, in a large enough sample, draft picks (not the players) should correlate with team talent levels and therefore wins too.

Are draft pick values non-significant because they don't matter, or because we don't have enough data? Consider M-T values in 2005 as a case study. The coefficient for M-T value without accounting for previous year's wins is 0.86 wins per million $ of surplus value. The coefficient drops to 0.24 when accounting for previous year's wins, but we cannot have much confidence in that number because it is not significant (p=0.42).

Example 1

In 2005, the team with the most M-T value was TEN with $5.3 million in surplus. The team with the least M-T value was NYG with $2.2 million. (The average was $3.7 million.) That's a difference of $3.1 million between the top and bottom teams.

Using the 0.24 wins/$value coefficent, we get:
3.1 x 0.24 = 0.75 wins.

Between the top team and bottom team in terms of M-T draft value, we could estimate a difference of 0.75 wins. That's a very rough estimate because of the lack of statistical significance, but it is significant in NFL terms. 0.75 games would be a lot in a 16 game season.

Example 2

Let's do the same drill with conventional draft points in the same year, 2005. Without considering the previous year's record, the coefficient for draft points was 0.0022 wins/draft pt. The coefficient drops to 0.00040 when accounting for previous year's wins, but it is not signficant (p=0.36).

In 2005, the team with the most draft points was SF with 4049 and the team with the least was BUF with 603 for a difference of 3446. The average that year was 1898 points.

Using the 0.00040 wins/pt coefficient, we get:
0.00040 x 3446 = 1.37 wins.

1.37 wins is very significant in NFL terms in a 16-game season, especially considering that the difference between 9-7 and 7-9 sometimes means going to the playoffs or firing the head coach. We just can't have much confidence that the real number is 1.37 without more data.

Conclusion

This leads me to suspect that conventional draft points, M-T values, or other measures of draft "capital" are probably important to a team's fortunes. I simply don't have a database large enough (yet) to determine a significant coefficient. But in the case studies above give us an idea of the likely potential magnitude of the effects. Admittedly, it is possible that the effects are zero or near-zero (or even negative). There is a third possibility--that draft picks do matter, but neither conventional draft points nor M-T surplus are good measures of their value. Another system of values, one that is calibrated to the effect the picks have on wins, might prove to be the best method of valuing draft picks.

The effect of the previous year's wins is powerful. The constant churn of poor teams improving and good teams getting worse dominates the effect of either draft variable. Ignoring draft variables, teams with 3 wins tend to improve to 7 wins the following year. Teams with 12 wins one year tend to stumble to 9 wins the next. I suspect this is partially why it is difficult to distill the true effect on wins of draft picks, assuming there is one.

The reasons for the churn may include strength of schedule, salary cap considerations, and the draft. But luck and other mathematical reasons may also be at work. I plan to address this in a later post.

Do Draft Picks Equate to Wins? 3

Does the Massey-Thaler (M-T) surplus value of picks translate into wins? I repeated the analysis I did with conventional draft points.

First, I performed a simple single-variable regression with "DeltaWins" (the change in team wins from the previous year) as the dependent variable and M-T surplus value as the independent variable. M-T surplus was significant with an r-squared of 0.03. The coefficient was 0.86, which means that the difference between the #1 team and the #32 team in the draft equates to 0.34 wins.

This is far less than the r-squared for draft surplus (0.26). But keep in mind, M-T surplus is not as directly related to previous year's record as draft points. We already know that a team's previous year's record is dominantly predictive of the following year's win total, regardless of draft points.

To account for the other factors that produce the tendency for poor teams to improve and good teams to get worse, the previous year's wins was added to the model. M-T Value is no longer significant (p=0.42) and last year's wins (LASTWINS) dominates. Adjusted r-squared = 0.38, which was the same as for conventional points.

I ran several more regressions and got very similar results to the conventional draft point models. Adding various combinations of M-T values, previous year values, and cumulative values did not improve the r-squared found by using last year's wins alone, which is about 0.37. Below is the table of results. Asterisks denote significance and negative coefficents are in red.

If draft points don't matter (compared to other factors), then M-T values matter even less.

Do Draft Picks Equate to Wins? 2

Teams that have losing records tend to improve and teams that have winning records tend to get worse. There may be several reasons including the draft, but scheduling and other effects may also be factors. This isn't any revelation, except that it severely complicates the question of whether draft picks can be converted into wins.

My first regression was a simple single-variable model that estimated net wins based on a teams conventional draft points in the immediately preceeding draft. This time, the model will include the teams' previous year's record to account for the other factors that explain the tendency for teams to regress to the mean--better teams get worse, bad teams get better.

Because draft points are mostly a function of a team's previous record, I expected a large amount of overlap in their effect on future wins. To really isolate and understand each variable's effect on wins, it would take some creative "differential" regression--we need to run several different regressions and compare the results.

Including both draft points and last year's win totals in the model reveals the following. Last year's record is highly significant and the coefficient is negative (as expected--more wins last year means less this year). Draft points, however, is not significant (p=0.36) but still postitive. The adjusted r-squared is 0.38.


Regressing last year's wins onto the following year's wins (without any draft variables) results in a virtually identical r-squared of 0.38. This means that draft points do not contribute any significant predictive power to the model. The other factors that explain why teams regress to the mean dominate the effect of the draft. This is strong evidence that draft picks don't matter, at least when compared to other explanations for improvements in team wins.

I conducted numerous other regressions. For example, I used draft points from 1 and 2 years preceeding the current year, theorizing that draft picks require a year or two to mature before impacting a team's record. I also used the cumulative draft points from the preceeding years. Curiously, draft points from 1 year ago was significant at the p=0.10 level, but its coefficient was negative. In other words, teams with high picks the year preceeding the season in question tended to get worse.

The table below summarizes many of the regression models I ran. Each row is one model. Significance is denoted by asterisks (p=0.10, 0.050, and 0.01 levels). Negative coefficients are in red.



No matter how it's sliced, previous year wins dominates the model.

Do Draft Picks Equate to Wins? 1

My theory was that yes, more and higher draft picks typically mean increased success to their team. Keep in mind I am talking about the draft picks in the abstract--not necessarily the players themselves but simply the selection positions. (Credit goes to Doug Drinen who originally attempted to use regression to test the connection between draft points and wins.)

To test the theory, I created a variable called DELTAWINS which is change in total wins from year to year for each team from 2002-06. This was my dependent variable. For my independent variables I used conventional draft pick point values. In other words, I used each team's total draft points from the immediately preceeding draft to see if it significantly correlated with an improvement in each team's season win total.

Simply put, higher draft picks--and more of them--should lead to more wins the following year. If so, the regression coefficient for conventional draft pick point values would be expected to be positive and significant. Actually, the coefficient is 0.0022 and is strongly significant, with an r-squared of 0.26. (The constant for the equation is -4.2.)

The team with the #1 pick in each round, the team with the worst record, has a total of 3957 points. It would be expected to improve by:

-4.2 + 0.0022 x 3957 = 4.6 wins.

The team with the #32 pick in each round, the Super Bowl champion, has a total of 1020 points. It would be expected to get worse by:

-4.2 + 0.0022 x 1020 = -1.9 wins.

However, the regression has problems. First, the distribution of draft points is not normal--a bell curve. (Someone who points this out is not normal either, I admit.) It is a gamma distribution. One of the prerequisites for linear regression is that the variables's distributions are approximately normal.

For now, however, we'll ignore that problem because there is a bigger one. NFL teams with losing records tend to improve and ones with winning records tend to get worse. The draft may be one reason, as my hypothesis states, but there are others such as "strength of schedule" or salary cap boom/bust cycles--more on the other possible reasons later. For now, we'll accept that the model needs to account for a teams previous W-L record.

Robust M-T Draft Team Comparisons

Previously, I ranked teams in terms of how well they converted conventional draft points into surplus performance value as calculated by the Massey-Thaler (M-T) study. Teams were ranked by draft points, then ranked by M-T value. The difference between each team's rankings indicates how well each team played the Massey-Thaler game.

Comparing ordinal rankings is not the most sophisticated analysis. Statisically speaking, continuous variables are more robust than ordinal variables for analyzing correlations.

So to see if a team, or teams, stood out at maximizing M-T surplus value, I regressed conventional draft points onto M-T value. In other words, I performed a simple regression with draft points as the independent variable and M-T value as the dependent variable. The sample includes the '02-'06 drafts.

The "residual values" of the regression are the differences between the predicted team M-T values and actual M-T values. If teams did no trading, and made no effort to increase M-T value, then the residuals should be zero, or close to zero. If a team made an effort to maximize M-T value then the residual for that team would be positive. Negative residuals indicate a team has "squandered" value in M-T terms.

Below is the table of residuals. For comparison, the table includes the difference in conventional points ranking and M-T value ranking I previously used to compare teams. Also listed are the team wins for the sample time period '02-'06.

The Titans and Rams are at the top of the list, while the Redskins again are at the bottom. There seems to be a drop-off from Tennessee and St. Louis to the rest of the pack. Perhaps they are teams that have struggled with salary cap issues during the time period, then were forced to makee draft pick trades to minimize the cap hit of their picks.

The average number of wins for the period is 40. The big question is 'how do team wins correlate with M-T value over the same time period?' The correlation coefficient is 0.44 (r-squared = 0.19). About 20% of the variance in wins can be accounted for by Massey-Thaler surplus value.

This is very remarkable. Conventional draft points should correlate very negatively with team wins. The more a team loses, the higher the picks it gets. In fact, the correlation between team wins and conventional draft points is -0.65.

M-T values are not as directly related to a team's previous record as conventional draft points. However, the losing teams with the higher picks still start with more M-T value than the winning teams with later picks. Therefore, if there is no "M-T phenomenon," then we'd expect the correlation between M-T "effort" and wins to be weaker than between conventional draft points and wins, but also decidedly negative.

Instead, we find that the correlation is solidly and significantly postive. (The significance threshold is 0.15.)

Overpaid Free Agent Positions?

In the last couple posts, the relative draft value of each position was calculated by summing the conventional draft points for each position. The same can be done with Massey-Thaler draft surplus value for each position. Here is the result:

RB, WR, TE, and CB top the list. But what does that actually mean? It's not as clear as draft point value. The point value from the conventional draft chart represents the draft "capital" spent to select a player. But the M-T surplus value represents the typical performance of the pick above his contract cost. The value of the pick's performance is based on the cost of a free agent with equivalent performance.

So what the table above actually shows is how each draft position overperforms compared to free agents at the same position. So the positions at the top of the list are the positions at which free agents are the most overpaid relative to draft picks.

Take RB for example, the position at the very top of the list. The data basically says that teams are probably better served by employing drafted RBs rather than free agent RBs. One example that comes to mind is the recent experience of the Colts. Marshall Faulk was an excellent player throughout his initial draft contract but the Colts did not resign him. They drafted Egderrin James who performed practically at the same level, and for half the price. When James' contract came up, the Colts allowed him to leave and replaced him with another draft pick--Joseph Addai, who was more than talented enough to help the team win a championship.

The same might be said for WRs, CBs, and TEs.

"Day 1" Draft Values by Position

In the last post, I calculated the value that NFL teams place on each position during the draft. The analysis accounted for the number of players required at each position in terms of players on the field (1 QB, 2 WRs, etc.) and in terms of roster composition (3 QBs, 6 WRs). I considered all 7 rounds of the draft from '99-'06. But "impact" players, those who are most likely to become starters and help win games, typically come from the first 3 rounds, rare exceptions excluded. So I performed the same analysis, but this time only considered players taken on "Day 1" of the draft.

The first table ranks each position by 1st-day draft points per player on the field. For comparison, I also list the relative ranking of each position per player on the field from all 7 rounds.

The second table ranks each position by 1st-day draft points per roster composition. Again, I also list the relative ranking per roster composition from all 7 rounds.


There is no difference in the relative rankings of the positions between day 1 and day 2. That should have been obvious to me from the outset because the value of picks drops off so sharply from the first round.

How NFL Teams Value Positions

Using the conventional draft value point chart, we can evaluate the relative weight that teams place on each position. By summing the draft points "spent" on players at certain positions we can rank the relative importance of each position in the draft. Here is a list of each position and how many total draft points have been spent on them based on the '99-'06 drafts for all 7 rounds.

It's an interesting list because high-profile positions such as QB or RB are not at the top of the list as we'd expect. Many other analysts make a chart like this each year, but there is a flaw with ranking positions this way.

A far better measure of the relative value placed on each position in the draft would be one that takes into account the number of players needed at each position. We can allow for the difference in the number of players needed in two ways. First, we can divide total points by the number of players of each position typically on a roster. Alternatively, we can divide by the number of players of each position typically on the field.

Here is the table of position draft pick value per player on the field:

Here is the table of position draft pick value per player on a typical roster:


Expectedly, "singular" positions such as QB and RB move up the lists because there are fewer of them required and "multiple" positions such as WR or CB move down the list. A lot of conclusions can be drawn from these tables, but what surprises me most is that RBs are valued as highly as QBs. Conventional wisdom says that solid QBs are so much harder to find than solid RBs, but it appears teams choose to spend roughly the same amount of draft points on those positions.

Perhaps the "typical" roster has more than 3 RBs or less than 3 QBs, but I'd suspect the true average number of roster spots isn't that far off. Plus, we get the same result when accounting for players on the field, and it's been a long time since we've seen 2 RBs on the field simultaneously.