That's the question that Cade Massey and Richard Thaler answered in 2005 in their landmark paper
. Their research findings demonstrated how the very top picks in the draft were overvalued relative to picks in the bottom of the first and top of the second round. Because the salaries of the very top picks climbed so high, the marginal surplus value of picks further down the draft board exceeded that of the picks at the top of the first round. The paper revealed numerous other lessons for team GMs, the most important may have been the folly of trading up, and paying a high price to do so. The authors
, and the main results now reflect data through the 2004 draft. But the new CBA radically altered the draft equation, and it's time to see how things have changed.
The Massey-Thaler paper covers several different aspects of how teams draft, and it's not easy to summarize but I'll try. It reveals that team decision-makers are overconfident in their ability to predict which players will turn out to be more productive, at least within a reasonable range of selection. In other words, GMs and scouts can distinguish the very best prospects from the replacement-level guys, but it's virtually impossible to confidently distinguish,say, the second-best prospect from the third or fourth-best at a particular position.
Based on a composite measure of player performance including games, starts, and top awards, players of the same position selected earlier in the draft are only marginally more likely to turn out better than the player taken immediately following. In other words, the first player taken at a certain position is only slightly more likely to turn out to be a better pro than the second player taken at that position, who in turn is only slightly more likely to turn out better than the third player taken at that position, and so on.
Perhaps the most meaningful aspect of the paper is that it proposed a completely new way of thinking about draft pick valuation. In an environment unconstrained by a salary cap, what matters in a draft pick is absolute player value. But in a salary-cap constrained environment, what truly matters is the surplus value
the draftee offers.
Surplus value is not an easy concept to explain. Say you're hungry and you walk by a fruit stand with apples priced at $1. You'd gladly pay $1 and, in fact, you'd pay up to $1.25 for the apple before you were indifferent between having the apple or not. The apple cost the vendor $0.75 to bring the apple to market, so he'd be happy to sell it to you for anything above that price. If you buy the apple for $1, you've created $0.50 of value, half of which you gain and half of which goes to the vendor. Win-win. You just got $0.25 of surplus value.
The surplus value of a draft pick works in a similar but not exactly the same way. It's the difference between what you pay for performance and what you'd be willing to pay for performance. Let's say the 1st pick of the draft offers you 10 "value units" and costs you $6M/yr. To obtain 10 value units in the Free Agent (FA) market, on average, say it might cost you cost $7M. That #1 pick therefore offers $1M of surplus because you would (or should) have been willing to pay up to the $7M price.
In the Massey-Thaler model, maximizing surplus value is the goal. It turns out the the very top picks in the draft were paid so highly that their surplus value was relatively modest compared to selections in the bottom of the 1st and top of the second round. That's the Loser's Curse
: Teams with the worst records were saddled with top picks with high salaries and low surplus, while the best teams reaped picks with slightly lower performance but much higher surplus value.
Now, there are some implicit assumptions in the Massey-Thaler model. For starters, player value to a team is additive. That's almost certainly not true, especially for a unique position like QB, but that doesn't mean we should dismiss the research, not at all. Reality may not be too far off from additive to invalidate the model's lessons, especially for positions other than QB.
Here's the main take-away from the research. Performance value
is defined as the salary cap price of a FA with equivalent expected value to the draft pick.
The paper's findings underscored how the very top picks were overvalued. There are numerous lessons for GMs and other decision makers. First and foremost is...never trade up. (And contrapositively, always trade down.)
Times have changed. Under the old CBA, a #1 pick like Matthew Stafford could command a $12M annual guaranteed salary over 6 years. Now, the maximum salary (in 2015 cap dollars) for the overall #1 pick is half that, and the rookie draftee pay scale has been flattened across the board. The flattening of the pay scale means that the relative surplus value of the very top picks should be higher than under the previous CBA.Methodology
Fortunately, we don't have to wait for a career's worth of seasons before we evaluate draft value under the new CBA. Draftee contract salaries are specified as a function of draft pick order--they're known certainties, something quite unusual in the world of sports analytics. Here's how we'll re-evaluate draft value.
We need to know 3 things:
|1. The salary cap hit of draft picks at each draft pick 1…256
|2. The expected performance level of each draft pick 1…256
|3. The estimated salary cap hit of a FA for a given performance level.
|The surplus value is therefore: Surplus Value = f (Pi) - Ci
||for each pick i
In plain language, (1) we need to know the cap hit of draftees for each pick #. We already know that from the CBA. We also need to know (2) what is the expected level of performance for players selected at each pick #. We can get that from regressing recent historical data. And lastly, (3) we need to know what each level of performance costs on the free agent market. We can also get that from fitting a curve to historical data. In case you're confused about where the i came from, that's just an index signifying which pick # we're evaluating.
The difference between the cap hit of the pick and the cap hit of a FA with equivalent expected performance is the surplus value of that pick.
For the measure of performance value, I'm going use Approximate Value (AV), Pro-Football-Reference.com's overall measure of player value. AV is similar in many ways to the measure of performance used by Massey-Thaler, specifically that it uses individual metrics such as game appearances, starts, and awards such as Pro Bowl selections to create a composite score. AV works differently in that it uses those metrics to apportion team success to players. You can get a full explanation of how it works here
The other main ingredient of the methodology is cap hit. That's the real cost of a player to a team. Cap numbers are complex, and in any single year for an individual player it may not be an accurate representation of long-term player cost, but in aggregate it is. Like other financial considerations, NFL salary caps are subject to inflation. The team cap limit tends to increase year over year, so a player's cap number in nominal terms is not the same in real terms. In other words, a $5M cap hit in 2006 is not as costly as a $5M hit in 2015, because team salary caps have increased. To account for this inflation, all salary cap values used in this study are adjusted to 2015 cap dollars.
All data in this study is from the 2006-2014 seasons.
The cost of each draft pick is fixed by the CBA. Here is the annual cost of each draftee by pick #.
As we would expect the very top of the draft features a steep drop-off and then the pay curve flattens for players taken in the later rounds. Because I'm using stated salaries from the CBA and not actual salary averages, I adjusted the salary amounts according to how likely each pick is to make the roster. This doesn't affect picks in the top rounds, but increasingly makes a difference in the bottom half of the draft.
The next step is to determine the expected performance value at each pick #. The average per-year AV from the #1 pick through Mr. Irrelevant looks like this. A logarithmic curve fits the data well.
[One interesting aside is to compare the performance curve and the pay curve for each pick #. The chart below shows that annual pay decreases more sharply than expected performance. This suggests that later picks represent bargains in terms of performance-to-cost efficiency. But keep in mind that efficiency is not the primary goal. ]
[This next chart highlights the efficiency concept, which plots the performance-to-cost ratio for each pick number. It shows that the peak cost efficiency in the draft is found at the bottom of the 2nd and top of the 3rd round. But again, pure efficiency is not how to build an NFL team in a salary cap environment. Filling your roster with nothing but 3rd-rounders would be cheap, but would create a woeful team and leave $70M of unused salary cap space. Efficiency is not the goal. Maximizing total player value subject to the cap is the objective, and the concept of surplus value is instructive toward that end.]
The third ingredient of the meal is the expected cost of a veteran free agent player for a given performance value. To isolate only free agent veterans, I filtered out any player in his 1st through 5th years. Once a player is in his 6th or later year, he is no longer under exclusive rights. There are some exceptions for 1st round picks in 6th-year option years under the old CBA, but they are relatively rare. The good ones had often already signed FA-worthy extensions and the bad ones didn't have their options picked up.
The next chart shows the expected cap hit of a FA at each level of performance value. The resulting plot fits a power law curve best, which makes sense. Power law functions (of the form y=axb
) are common in pay distributions where top performers are increasingly steep compensation
. Other related phenomena in pro football, such as Pro Bowl selections or career longevity, tend to follow the same curve. But for the purposes of this study, the range between 1 and 10 AV, which is the range of possible expected AV values for draft picks, the curve best fit by a simple line. Because I'm interested in accurate empirical estimations and not a theoretic model, I'll use the linear estimate.
Here are how the salaries of draftees and equivalently performing FAs compare. To obtain a given level of performance, a team pays a premium for a FA. There could be reasons for that premium, but it is considerably large.
Now we have everything we need: the pay scale for rookies, their expected performance levels, and the cost of equivalently performing FAs. The difference between what a particular level of performance costs for FAs and for draftees is the surplus value. Here's the surplus value at each pick #.
The green line is the surplus value. It ranges from under 1/2 a million to a peak of over $2M through the second round. The irregularities/discontinuities in the curve are due to the draftee salaries. The abrupt uptick in surplus value is at pick #33, where the second round begins. The general results are congruent with Massey-Thaler's original surplus value curve, but the particular magnitudes of the surplus are considerably different.
I expected that the new CBA would result in much more surplus value at the very top of the draft, but that's not the case. The timeframe of the Massey-Thaler paper did not include the Stafford-Bradford years, where very top draftee salaries exploded. It seems that the new CBA restored the balance of an earlier era, the era covered by Massey-Thaler, rather than radically altering the landscape. It seems that teams are now paying nearly market salaries for the very top picks.
These new results indicate much higher surplus value for "second day" picks of the 2nd and 3rd rounds than previously found. The causes of the difference are that the "FA Cost" curve (red curve above) is slightly flatter than before (compared to Massey-Thaler's "Performance Value" curve), and that the salaries of mid-round picks are noticeably depressed. The combination of both effects results in greater surplus.
Some self-criticisms: For the salaries of FA veterans, I've relied on years that span both the current and previous CBA. The assumption is that the new CBA doesn't significantly affect FA pay. But it would, even if indirectly. If less money is spent on rookie contracts, it leaves a bigger share of the salary cap pie for veterans. This might increase the FA Cost values slightly and result in slightly higher surplus values.
Also, the entire study is highly sensitive to one modeling decision. In the Massey-Thaler methodology, FA Cost is modeled as a function of Performance Value. In my version here, that means that FA Cost = f
(AV) + error. In other words, to get the FA Cost-equivalent curve (red above), I look at the average cap hits at each level of AV and fit a curve. I'll call that model 1. However, if do things slightly differently and assume Performance Value is a function of pay, which I'll call model 2, then the results are completely different. Model 2 would be a curve fit on AV= g
(FA Cost) + error. To get the FA Cost for a given amount of AV, we would say FA Cost = g-1
(AV). That's the inverse of the performance curve, which just means we're flipping the x and y axes. There's a good reason for doing it that way, but I'll have to save that discussion for a different post.