I looked at salary data from 2000 through 2009 courtesy of USA Today and compared it with stats like Win Probability Added (WPA) and Expected Points Added (EPA). For those not familiar, WPA measures the impact a player has on his team's fortunes in terms of wins. EPA measure his impact in terms of net point differential. I looked at other stats too, but EPA and WPA fit very nicely with salary, at least for QBs.
Specifically, I wanted to discover what teams are willing to pay in exchange for expected levels of performance. To find out, I plotted the salary cap values of QBs against their performance. First, I made an adjustment for year. Between 2000 and 2009, QB salaries have steadily increased. By 2009, they were about double what they were to start the decade. This fits nicely with overall team cap numbers, which approximately doubled between 2000 and 2009.
QB performance has inflated too, but I haven't yet added a correction. QB performance has increased a total of about 10% since 2000, which pales in comparison to salary. In the future I'll add a correction and it might sweeten up the results a bit.
NFL contracts are notoriously complex, with signing bonuses, guarantees for both performance and injury, roster bonuses, and performance incentives. But none of that matters to me. I just want to know how much of a team's payroll under the cap--its most precious resource--is it willing to spend. How much are they spending, and how much are they getting back? For now, I'm looking at all big-money players, whether they're draft picks or free agents.
Here is how the relationship between salary and performance shakes out. The top graph plots EPA vs. cap hit, and the second graph plots WPA vs. cap hit broken out by season. The plot filters out QBs that earned less than approximately $2M in adjusted salary.
The best-fit line is pretty shallow and the correlation isn't terribly strong (r = 0.26). Despite a near-zero p-value, it there is a great deal of variability in a QB's year-to-year performance and how it relates to his pay. But that's something we should expect given all the things that can happen over the course of a season and how dependent any QB is on the team around him. This plot suggests teams are getting back about 7.7 EPA and 0.17 WPA per season, for every million dollars they spend on a QB (in 2000 cap dollars).
QB performance is also highly dependent on teammates, opponents, schemes, injuries, etc. Implicit in any analysis like this is the assumption that over enough time and enough cases, those factors tend to even out. After all, the vast majority of QBs are signed to multi-year contracts, when year-to-year factors do tend to even out.
With that in mind, I plotted the data in a different way, this time aggregating career salary and career performance for the top paid QBs over the past decade. For example, in the 2000 through 2009 seasons, Brett Favre cost $61M in cap space, and returned 733 EPA and 13 WPA. I believe this method can better tell us what teams are getting in return for their cap dollars. QBs who cost less than $10M over the entire period were excluded.
This method provides a stronger correlation between what teams are paying and what they're getting over time (r=0.76). In this light, QBs produce about 16.8 EPA and 0.39 WPA per season per million (in 2000 dollars). The 2011 cap is almost exactly double what it was in 2000, so we could safely translate those numbers into 8.4 EPA and 0.20 WPA per million.
There are other ways to skin this cat. We can look at per game or per play measures, but they all produce roughly the same equation.
You might be asking why I don't produce a complex regression based on a jumble of stats that include passing yards, sacks, sack yards, interceptions, fumbles. The answer is that I already have. The reason I can use a simple uni-variate (single variable) regression is that the EPA and WPA frameworks already incorporate nearly all the aspects of QB play and weight them in accordance with their effect on net point production (EPA) or win production (WPA).
The replacement level QB conveniently appears to be near zero EPA. The minimum salary has doubled over the past decade, but averages near $300k per year. Players at near that salary level average near zero EPA. Across all of football, zero EPA would be the average expected outcome for all plays. But average QB play is significantly higher than zero for a couple reasons. Passing, in general, provides positive net value compared to running. Also, several other non-plays which do not involve QBs, such as false starts and other penalties, tend to be negative. Zero EPA is therefore solidly below average for a QB. This season, zero EPA/replacement level performance was achieved by the likes of Rex Grossman, Matt Moore, and Mark Sanchez.
What would this mean for Drew Brees, who is scheduled to be a free agent after the season? Over the last six regular seasons, Brees has averaged 150 EPA. Ignoring age or injury risk and using a replacement value of zero EPA combined with the slope from the aggregate plot, the numbers suggest that the market for Brees would be nearly $18M per year in 2011 money. But with Brees' scorching hot 2011 season (250 EPA) fresh in everyone's memory, it wouldn't surprise me to see him sign for more.
There is a lot of work still to be done before any of these numbers are definitive, but this kind of analysis opens a whole new window into player analysis.
One fun game to play is most-overpaid/most underpaid. They guys above the best fit line are the bargains, and the guys below the line are the busts.
It would be interesting to see the same analysis for RBs.
I always thought Jeff Garcia got a bad rap.
re: overpaid/underpaid QBs, I don't think you can just use this study to determine it. If Brees signs for $20M but is only worth $18M, you wouldn't argue that he's overpaid and/or not worth his contract because the Saints aren't able to just go sign a replacement. Great QBs are such a scarce (yet crucial) commodity that it's probably worth keeping them at any semi-reasonable cost, no?
Kos
In that case you'd actually need some other numbers to work out whether that $20m is best spent on Brees, or whether they should get a replacement QB and spend the $20m on e.g. better defenders
Great analysis Brian. It's great to see the allocation of money.
Kos- I think what your describing is a consistent pattern in salary cap sports where there is a limited resources. There are only so many players that can generate high EPAs / WPAs. The best strategy is to slightly over pay for positions responsible for high EPA / WPA variability, and underpay for positions with little EPA / WPA variance.
I was concerned after looking at the first pair of graphs. That's a classic "you can draw a line through a cloud of dots" graph.
The second pair is pretty good. The outliers are the most interesting: Jeff Garcia was underpaid, Brett Favre and Donovan McNabb were slightly overpaid, and Joey Harrington was just plain awful. And I think it's time to get on a phone and call Mark Brunell's agent! He's still got a job!
Part of a QB's success is due to the other players on the offense (and how much he is used), and part is due to luck.
So if you strictly follow this scale, you are almost certainly overpaying for "good" seasons, like the one Brees had. That is, it's incredibly unlikely that Brees continues to be as good as he has been.
You should really adjust for his likely decline. It would be interesting to compute how much this tends to be in the NFL for QBs, but I would guess something like 10-20%/yr is typical.
A problem with the second analysis (which aggregates the whole decade) is that QBs get paid based in part on past performance. For instance, Marc Bulger performed very well from 2002-2006, and was signed for cheap. Then in the 2007 offseason he signed a big contract based on his past performance, and he proceeded to flop (he scored negative EPA & WPA his next 3 years). If you compare his career salary (most of which came after 2007) with his career EPA or WPA ([more than] all of which came before 2007) then he falls right on the regression line, but he was really wildly underpaid for 5 years and then given a new contract which wildly overpaid him for 3 years.
Another way to divide things up is to use one data point for each contract. The x-axis would be the cap hit per year over the life of that contract, and the y-axis would be the EPA/year (or WPA/year).
That's true. I'd need much more time than I currently have to parse out all the individual contract data. This method just tells us what teams spend and what they should expect to get.
Ultimately, I'd like to identify rookie contracts, FA contracts, franchise tag contracts, guaranteed money, etc.
As a Jets fan, please tell me where the jets can get a "replacement level" QB so they can cut him, and use the money on a Right Tackle, cover safety, and Line Backers.
Anonymous Jets fan,
It's called free agency and the NFL draft. You should look into it. All these guys were available this offseason and had a higher EPA/play than Sanchez - Matt Moore, Tavaris Jackson, Matt Hasselbeck, Dan Orlovsky, and TJ Yates. And that's not including guys who were available recently or for a bit more like Palmer, Grossman, McNabb, and Campbell.
re: overpaid/underpaid QBs, I don't think you can just use this study to determine it. If Brees signs for $20M but is only worth $18M you wouldn't argue that he's overpaid and ...
As to over/underpaid, a relevant consideration is what form the return on cap expenditure takes. Owning an NFL team is a business, so to a significant extent the real return is money, business revenue, not wins.
This is much clearer in MLB, which has no salary cap, and where the financial return on a player for teams like the Yankees and Red Sox in big markets may be much larger than for small market teams, so over paying per win can make perfect sense for them. It is also clear in the NBA, where teams plainly over-pay for big scorers compared to the wins they produce, because fans like big scorers.
This graph impresses by indicating that in the NFL QB pricing pretty much is consistent for all, notwithstanding. Apparently the cap (and TV revenue sharing), by putting teams on a more equalized financial playing field, more closely tie winning with financial returns.
Still, I wouldn't be surprised if Peyton being so far above the line -- being significantly overpaid for his his performance, as great as it is -- is because he as an individual brings in more revenue for his team than any other QB does.
Great QBs are such a scarce (yet crucial) commodity that it's probably worth keeping them at any semi-reasonable cost, no?
If the object is to win, then "no" as to overpaying them for the wins they produce, because the money saved by using another non-overpaid per win QB could be invested to produce more wins via other positions. The over-paying for Peyton per win has I think clearly compromised the Colts defense over the years -- Polian said publicly that they adopted the version of the Tampa-2 D they used because it can be operated at less cap cost than any other system -- and with a better D they might have collected more than one trophy during the Peyton era.
It also could be, given the way the NFL ever-more markets football games as a QB versus QB celebrity match-up events, that QBs as a group are over-paid per win compared to other players because they are the players who bring the fan revenue in, just as the NBA consistently over-pays its big scorers per win they produce, and under-pays its other players.
But to know that, we'd have to have a chart showing how much QBs get paid per win compared to how much other players do. But there's too little data to tell, so far. More charts, please!
Jim, you've misread the charts. Peyton, Brady, Brees and everyone else above the line are UNDERpaid. For instance, Brees has made approximately $25 million, when his performance has warranted $55 million.
In fact, you could make the case that Brees has been the Most Valuable Quarterback in the NFL when comparing production to cost.
With the whole Gladiator effect, I would expect top-level QBs to be worth more than a straight-line slope would give them credit for. They've looked into this on fangraphs.com, where for example, Pujols is worth 7.0 WAR (wins above replacement). You can't just pay two guys to get you 3.5 WAR and get the same result. Therefore, you might expect the true curve to be concave upward.
So I would think that the best QBs are worth a little more.
Jim, you've misread the charts.
I spend a professional lifetime reading charts with the $$$ on the other axis, and this happens. Behavioral conditioning or early Alzheimer's? My bad.
Never mind.
Awesome analysis! I've been hoping someone would do for football something like what Victor Wang did for baseball, and this is a great start.
Very cool post.
And James- excellent call, but poor Jeff Garcia. Not only did he get a bad rap, he apparently got $30M worth of it.
I always felt bad for the guy. He was a scrapper who never found the endearment of his announcer, was at least mildly successful everywhere he went (except Cleveland, where larger forces are to blame), but also seemed oh-so-replaceable to whomever was in charge.
In related news, I'd love to see this from the start of Brett Favre's career. I just like to be reminded of how good he was, before he more or less stopped caring about double coverage.
Hey Brian, or any of the other bloggers,
Can you release a full list of quarterback salaries for 2012.....I am currently working on my thesis and could really use that data. I have found some numbers here and there but it is hard to form a list from the data of a range of sources (some of which might not be reliable). Thanks a lot for your help!
Jon--Check out spotrac.com.
Why not use % of Cap consumed instead of $ amounts? I think that would scale better over the years than having to continuously adjust
So, BOTTOM LINE: teams will pay players 5 million per win added? That is, .2 wpa * X = 1 Win, solve and you find x = $5 MM
You say QB $/win might be/is different than other position $/win.
If QB $/win is different from other positions then I think THAT IS YOUR KEY FINDING.
Why? Because, a win is a win. It doesn't matter what position you get it from. A positional difference in $/win represents a misallocation of millions per year by GMs.
Thanks Tom for recognize that "wins" is not a QB statistic.