2009 Final Quarterback WPA and EPA

Here are the final numbers for quarterback Win Probability Added (WPA) and Expected Points Added (EPA) for the 2009 season. These numbers include playoff performance. I've posted these stats previously for selected players, but this is the first time I've published a comprehensive list. This is the first time I've truly had confidence in the absolute values of the WPA stats. Previously, they really could only be relied on for relative comparisons between players.

You might notice that the WPA and EPA stat totals vary from the previous posts. That's because I've fixed some discrepancies in both systems. For the WP model, I've recalibrated the parameters so that the grand sum of all plays in all seasons add up to zero. Previously, there was a tiny positive bias on every play that when added up over hundreds of plays made a significant difference. Additionally, the EPA model now excludes final plays of each half. Previously, EPA counted the final play of each half as a very negative event. EPA calculates the difference between the net point value before the play and the net point value after the play. Because the EP value at the end of a half is always zero, my system was heavily penalizing players who participated in the these plays. This affected QBs in particular because many final plays are kneel-downs. There are still some tweaks and fixes to be made, but they are relatively minor.

One other thing to note up front is that the payoff for passing is predominantly better than for running in terms of both WPA and EPA. I've published extensively on the apparent advantage of passing over running based on the EPA model, but I have yet to publish any research based on the WP model. Now that it's been calibrated properly, we have a way to discern the relative value of passing vs. running while taking into account the considerations of time and score. Much of the value of running is obviously when a team with the lead can burn time off the clock, but that's a discussion for another article. The bottom line here is that the average QB will have significantly positive WPA and EPA totals. (Conversely, the average running back will have a negative totals.)

The first table below lists the top QBs, plus some other notable ones, ranked according to their 2009 WPA totals. Total EPA is also listed for each QB. The last column, Ratio, is something new. It's simply the ratio of a QB's WPA to his EPA. This is a rough measure of leverage, timing, or "clutchness." In other words, it measures how much win probability a QB squeezed out of his raw performance due to situational factors.

Teams with QBs who had a very high WPA/EPA ratio could be expected to regress next season. EPA is less context-dependent and therefore can be expected to be more repeatable. Guys like Shaun Hill, Bruce Gradkowski, and Mark Sanchez did more with less in 2009, but we shouldn't count on them to do the same in 2010. Trent Edwards has a very high ratio, but that can be discounted due to a very tiny EPA. (Someone with a perfectly-average zero EPA would have an infinite ratio.) I may need to come up with a more robust way to calculate this--suggestions are welcome.

Click on the headers to sort.

 Rank Team QB WPA EPA Ratio 1 18-P.Manning 9.07 208.6 0.043 2 17-P.Rivers 5.83 181.3 0.032 3 9-D.Brees 5.73 171.4 0.033 4 8-M.Schaub 5.20 121.9 0.043 5 4-B.Favre 4.27 168.1 0.025 6 12-A.Rodgers 4.25 161.6 0.026 7 7-B.Roethlisberger 3.92 124.3 0.032 8 9-T.Romo 3.80 97.2 0.039 9 13-K.Warner 3.19 80.8 0.040 10 9-C.Palmer 3.07 16.7 0.184 11 10-E.Manning 2.84 117.3 0.024 12 2-M.Ryan 2.77 78.8 0.035 13 12-T.Brady 2.50 111.9 0.022 14 5-J.Flacco 2.47 66.3 0.037 15 8-K.Orton 2.38 53.4 0.045 16 5-D.McNabb 2.28 73.8 0.031 17 10-V.Young 2.18 43.5 0.050 18 7-C.Henne 1.54 37.3 0.041 19 6-M.Sanchez 1.36 2.2 0.630 20 6-J.Cutler 1.34 12.6 0.106 21 5-B.Gradkowski 1.16 8.3 0.140 22 9-D.Garrard 0.87 16.8 0.052 23 3-M.Moore 0.70 30.7 0.023 24 9-M.Stafford 0.69 -51.2 -0.013 25 8-C.Redman 0.62 19.8 0.031 26 17-J.Campbell 0.49 12.5 0.039 27 13-S.Hill 0.46 0.6 0.767 28 4-K.Kolb 0.42 9.3 0.045 29 7-M.Leinart 0.22 -6.1 -0.036 38 10-B.Quinn 0.02 -22.7 -0.001 52 8-M.Hasselbeck -0.16 -30.1 0.005 58 5-T.Edwards -0.31 -0.1 3.100 67 2-J.Russell -0.51 -76.9 0.007 69 10-M.Bulger -0.61 -21.1 0.029 74 17-J.Delhomme -1.31 -24.8 0.053 75 11-A.Smith -1.43 -16.7 0.086 76 7-M.Cassel -1.47 -54.8 0.027 77 3-D.Anderson -1.66 -57.1 0.029

The second table lists each QB's WPA and EPA on a per-play basis.

 Rank Team QB WPA/P EPA/P 1 18-P.Manning 0.012 0.29 2 17-P.Rivers 0.010 0.30 3 9-D.Brees 0.009 0.26 4 8-M.Schaub 0.008 0.18 5 10-V.Young 0.007 0.14 6 7-B.Roethlisberger 0.006 0.20 7 4-B.Favre 0.006 0.24 8 5-B.Gradkowski 0.006 0.04 9 12-A.Rodgers 0.006 0.23 10 2-M.Ryan 0.006 0.16 11 9-C.Palmer 0.005 0.03 12 9-T.Romo 0.005 0.13 13 13-K.Warner 0.005 0.12 14 10-E.Manning 0.005 0.20 15 8-C.Redman 0.004 0.14 16 8-K.Orton 0.004 0.10 17 3-M.Moore 0.004 0.18 18 4-K.Kolb 0.004 0.09 19 5-J.Flacco 0.004 0.10 20 5-D.McNabb 0.004 0.12 21 12-T.Brady 0.004 0.16 22 7-C.Henne 0.003 0.07 23 7-M.Leinart 0.003 -0.08 24 6-M.Sanchez 0.003 0.00 25 13-S.Hill 0.002 0.00 26 6-J.Cutler 0.002 0.02 27 9-M.Stafford 0.002 -0.12 28 9-D.Garrard 0.001 0.03 29 17-J.Campbell 0.001 0.02 30 10-B.Quinn 0.000 -0.07 31 8-M.Hasselbeck 0.000 -0.06 32 2-J.Russell -0.002 -0.26 33 10-M.Bulger -0.002 -0.07 34 7-M.Cassel -0.002 -0.09 35 17-J.Delhomme -0.003 -0.07 36 11-A.Smith -0.004 -0.04 37 12-K.Boller -0.005 -0.22 38 3-D.Anderson -0.008 -0.27

23 Responses to “2009 Final Quarterback WPA and EPA”

1. Tarr says:

Doesn't the WP model already take into account the clock? That is, shouldn't the WPA calculation take into account that a 2 yard gain that runs 40 seconds off the clock can increase the probability of winning? Is 2nd-and-2 with an 8 point lead and 3 minutes remaining better or worse than 1st-and-10 with an 8 point lead and 3:40 remaining?

2. Brian Burke says:

Yes, definitely. The only difference now is that the model's output is calibrated so that the average play is (nearly) exactly zero WP.

Previously, it came out to +0.0003 or something like that. This would distort the relative value of play types. For example, say passes would end up being +.009 WPA/pass, and runs would come out to being +.001 WPA/run. That would be different than +.006 WPA/pass and -.002/run. Relatively, things look the same. But the absolute numbers look slightly different.

3. Borat says:

What is the frequency, Brain?

4. Anonymous says:

>>Teams with QBs who had a very high WPA/EPA ratio could be expected to regress next season.

Brian,
In looking at Mark Sanchez, have you done any analysis on how quarterbacks improve in their 2nd and thrid seasons? Sanchez has a 1.36 WPA and it must have been entirely due to the last game of the season against the Bengals and teh Playoffs. I'm hoping he improves quite a bit, cutting the INTs by at least 1/3 and increasing his completion percentage to the high 50's.

- Brandon

5. JKL says:

One way to do the ratio would be to convert the stats to something similar to the advanced passing tables on PFR. Find the average and standard deviations for WPA and EPA, and convert those to a number (average = 100), with those above average having a number greater than 100 and those below average are in double digits, so that all qb's are going to have a positive rating. Compare the ratings relative to the two stats. If a quarterback is a greater number of standard deviations away from the average in WPA than in EPA, it shows he leveraged his performance (or was lucky that it occurred at key times), relative to other quarterbacks.

6. Brian says:

As far as "clutch WPA", you could calculate a league-wide average ratio of WPA/EPA, and multiply this by a given quarterback's actual EPA. This would give you that QB's expected WPA if he were of "average clutchness". The difference between his actual WPA and his predicted WPA would give you something like "clutch WPA".

7. Brian says:

FYI, I got a league-wide ratio of 0.041, which would give Matt Stafford the most "clutch WPA" for earning 0.69 compared to the -2.1 WPA that we'd expect based on this -51.2 EPA, for a clutch WPA of 2.79. The lowest clutch WPA would belong to Favre, who earned only 4.27 WPA compared to the 6.88 we'd expect, for a clutch WPA of -2.61. Hmm.

8. Alex says:

Is 'clutchness' a positive attribute here? If a quarterback had the option of when to throw a touchdown, he would be better off according to WPA or WPA/EPA to throw it in the 4th quarter. But those touchdowns are really only useful if they come in high-leverage situations, which will be when his team is losing or about tied. It feels like this measure will be extra-generous to those teams who couldn't put the game out of reach early. On a related note, Peyton Manning apparently set the record for comeback wins this season; I wonder what the number is for Rivers, Brees, Favre, and the other top QBs here.

Hey Brian,

Could you quickly explain how EPA works when a score happens. For example, you have the ball 1st and 10 from the opponents 15. Your expected points is 4.39. So if you score, is it 6-4.37 or 7-4.37 or do you also have to take off .7 for the insuing kickoff? I fully understand EPA except for how it works on scoring plays. Same question concerning field goals.

Thanks Brian

10. Ian says:

I've never really understood the idea of 'clutch', especially in terms of skill players like QBs. If a QB throws 3 TDs in the first quarter then none for the rest of the game we say he started on fire then struggled. If they suck for 3 quarters then throw 3 TDs in the 4th, we say they are clutch. But both threw 3 TDs, so who's the better QB?

On Matt Stafford's 'clutch' rating you describe, is the problem there not simply that whenever Stafford threw a pick it's when his team had little chance of winning, so it cost them EP but hardly anything in WP terms? Perhaps what this really shows is that Stafford is better than the rest of his team. The positive WPA suggests he didn't cost them games, but having such a large minus EPA suggests he did cough up the ball a lot. If you're turning it over but it's not affecting your chances of winning that means you're turning it over when you've already lost.

On the other hand, if you had a positive EPA but negative WPA that would suggest you only throw garbage time TDs, and when the game is actually on the line you don't perform.

11. Brian says:

Hey, just to be clear, I'm not Brian Burke, just another fan named Brian, so those were just my quick calculations. I agree with all of Ian's points though. It seems like if you're good enough, you'd never have to be "clutch", because you'd always be up by 35 points, right?

12. Brian Burke says:

Jay-To answer your question, a TD play from a 1st and 15 would be worth 6.4 - 4.39 = 2.01 EPA.

The EP value of a TD is 6.4 points. That's the 7 for the TD minus the kickoff value of 0.6 EP.

Perfect Brian, thank you.

Is it difficult to measure field goals then at the end of a half? Since there are times when a field goal will come on first, second or third down, it might actually create negative values for the kicker because the spreadsheet shows higher expected values.

Thanks again for your great work and taking time to answer the questions.

14. Brian Burke says:

Good point, Jay. Right now EPA is just ignoring all last plays of the half, which is when those kicks would usually occur. But sometimes they don't, so I'll be on the lookout for those.

There are a number of other interesting decisions to be made which I'll address in the next post.

15. Anonymous says:

Josh Freeman????

16. Jim Glass says:

I've never really understood the idea of 'clutch', especially in terms of skill players like QBs. If a QB throws 3 TDs in the first quarter then none ...

Some true life examples may clarify the reality of "clutch play", probably better than a hypothetical.

~~~
Vinatieri is generally considered to be one of the best clutch kickers in the history of the National Football League. Nicknamed "Automatic Adam" for his accuracy, and "Iceman" for his incredible poise under pressure...
~~~~

... and then cites Vinatieri's last second 41-yd FG to win the Super Bowl game against Carolina as an example of his great clutch kicking.

It doesn't mention that Vinatieri earlier in that game missed 31-yd and 36-yd attempts. But if he had made those, so his team had been up by six at the end, then that last kick wouldn't have been historically "clutch" to seal his rep. The Pats might have just run a FB into the line to run out the clock instead.

"Automatic Adam" the "Iceman" hit 1 of 3 from an average distance of 36 in the Super Bowl.

Note how the same events also sealed Brady's rep as a great "clutch QB", as in the last one minute he marched the Pats near the length of the field to position The Iceman's winning FG. Pats fans consider that a big entry on his resume of clutch play.

But omitted from it is that just a few minutes earlier in the 4th Q, while winning and apparently driving to put away the game, Brady threw a pick on the Carolina 9. Carolina promptly went the distance the other way, so that very arguably was a 2-TD swing pick. If Brady doesn't throw that horrid pick, but instead just scores routinely from the 9, he doesn't get a chance to be "clutch hero" either.

In fact, unless Brady throws that pick from the 9 and Vinatieri misses from 31 and 36, neither gets to be a "clutch hero" with their great play under pressure. They cemented each other's reputation for greatness with those short misses and red zone pick.

Another ESPN Classic example: An entire generation has grown up watching the clip of Joe Montana, "Mr. Clutch" of his generation, throwing 'The Catch' to Dwight Clark in the last seconds to win that NFC Championship game, the embodiment of "clutch play" on video, forever.

Who remembers that Montana threw three interceptions earlier in that game? But if he hadn't thrown three picks, he'd probably never have been behind like that to show how he was the guy all could count on to never make a mistake in a big games in the clutch.

Thus "clutch play" is explained.

17. Jim Glass says:

As far as "clutch WPA", you could calculate a league-wide average ratio of WPA/EPA, and multiply this by a given quarterback's actual EPA...

Brian, I used your method (using a median ratio of 0.037) and also got Stafford as the "most clutch" QB. Spreadsheets agree!

The top 5 "most clutch" off mine...

1. Mark Stafford 2.6 WPA over expected for EPA
2. Carson Palmer 2.4
3. JaMarcus Russell 2.3
4. Petyon Manning 1.4
5. Mark Sanchez 1.3

Hmmmm... One who believes in "clutch play" would expect Peyton to be there.

But as to Russell and Stafford, maybe they hit some sort of lower bound, their EPAs were so negative their teams couldn't lose that many games? Palmer was on a team that made the playoffs, maybe he had a better year than most people thought. Sanchez I watched all year, after the first few games the coaches really put the handcuffs on him, wouldn't let him throw in any kind of demanding situation. So maybe he didn't make the bad plays in critical situations that could have hurt the team -- he had fewer "important bads" than his EPA would suggest. Sort of a coach-ordered "negative clutchiness"?

Altogether, quite a mix. As is the order working down the rest of the list. I don't see a whole lot of support for "clutch player" concept in it.

18. Ian says:

A similar idea to what others have done, but I've done a linear regression on the EPA/WPA data, giving WPA = 0.03 x EPA + 0.5. If you calculate the residual WPA for each player, you get a kind of 'effectiveness' ranking for wins added above/below average for a given EPA.

Top 5 are
P.Manning +2.4
C.Palmer +2.1
M.Stafford +1.7
J.Russell +1.2
M.Schaub +1.1

Bottom 5 are
A.Smith -1.4
B.Favre - 1.2
E.Manning -1.1
J.Delhomme -1.1

The bottom 5 are interesting. One of the common complaints about Brett Favre is that he pads his stats when the game is already won. For me, these numbers add more weight to that suspicion, as being bottom of this list means that the points added by that player aren't affecting the chances of the team winning. Seeing as the Vikings won 12 games it's unlikely that Favre threw a majority of his EPA when the Vikings WP was low. Much more likely that much of his EPA came when they were already ahead.

19. Jim Glass says:

Seeing as the Vikings won 12 games it's unlikely that Favre threw a majority of his EPA when the Vikings WP was low.

As to the existence of "clutch play" I'm a skeptic-to-cynic, and so expect measures of it to produce pretty much random results -- which a ranking mix of P. Manning, Palmer, Stafford, Russell, Schaub, Sanchez at the top ... A. Smith, Brady, Favre, Delhomme, at the bottom, pretty much looks like.

But beyond that I think you have a point about a systematic factor.

The QB is less of the team than many fans think, and than all the publicity hoopla about mano-a-mano QB match-ups would have them think. He's only one of 22 starting players, of maybe 35 who make up the real "quality" of the team. So even if he's worth four or five others, the other guys cause the bulk of the game results.

If a QB is on a good team, and the other players have him ahead on the score most of the time, his opportunity to make big game-turning winning plays will be reduced, biasing his WPA/EPA ratio down. (Favre, Brady at the bottom.) If he's on a bad team and the other players have him behind on the score most of the time it's the reverse, his opportunity to make big game-turning losing plays will be reduced, biasing his ratio up. (Stafford, Russell at the top)

OTOH, the Colts this year played seven 4-pts-or-fewer games, so Peyton had a lot more chances than most to make a WPA difference.

I couldn't prove this, but it seems plausible. For instance Stafford is at the top of all these lists, and look at his numbers:

Passes when behind: 265.
Passer rating in 4th Q within 7 points: 36 (!)

It doesn't look like he got his high WPA/EPA ratio by making good "clutch" plays in close games in the 4th.

20. Bigmouth says:

Brian, is there any support in football for a skill called "clutch" that translates from year to year?

21. Ian says:

Looks like the rank column is messed up in the first table. The values jump by increments greater than one toward the bottom.

22. Brian Burke says:

Ian-That's intentional. Those are the 'notable others' who weren't in the top 30 but I chose to include.

23. Brian Burke says:

Bigmouth-I would seriously doubt it. And the sample sizes in a football season are almost certainly too small to be able to tell. It may be worth a look though.