Luck and NFL Outcomes 1


INTRODUCTION


Over the past few weeks, I've been interested in the amount of luck in NFL outcomes. I was interested primarily because I wanted to know just how good a game prediction model can get. In other words, what's the theoretical best that a prediction model can do? 70% correct? 95% correct? I think I've stumbled upon the answer.

The very best computer models predict winners at only a 70-75% rate. But that's not saying much because a monkey could predict winners 50% of the time. A monkey who knows which team is the home team could be correct 58% of the time. Even the Las Vegas odds makers aren't much better. They're correct less than 65% of the time.

It got me thinking. If a team is the very best team in the NFL, why wouldn't it have a 100% chance of winning each game? Why aren't there lots of 16-win teams? I thought that there must be good deal of luck involved to prevent the #1 team in the league from winning more than 13 or 14 games each year. Otherwise, why wouldn't the best team win 16 games every year?

In this post, I'll compare the actual distribution of NFL season wins to the distribution of a league determined by pure luck. Next, I'll compare the actual distribution to a league that theoretically is based on pure skill. Then finally, I'll show how I mathematically synthesized those two comparisons to determine exactly how much of the NFL is really just luck.

WHAT I MEAN BY LUCK

I'm not talking about a freak gust of wind or a slick patch of turf at a critical time and place to alter the outcome of a game. Although things like that happen, I'm talking about a much more ordinary phenomenon. An example I've used before goes like this:

Consider a very simple example game. Assume both PIT and CLE each get 12 1st downs in a game against each other. PIT's 1st downs come as 6 separate bunches of 2 consecutive 1st downs followed by a punt. CLE's 1st downs come as 2 bunches of 6 consecutive 1st downs resulting in 2 TDs. CLE's remaining drives are all 3-and-outs followed by a solid punt. Each team performed equally well, but the random "bunching" of successful events gave CLE a 14-0 shutout.

The bunching effect doesn't have to be that extreme to make the difference in a game, but it illustrates my point. Natural and normal phenomena can conspire to overcome the difference between skill, talent, ability, strategy, and everything else that makes one team "better" than another.

For more on how I define luck, see this post.

A PURE LUCK LEAGUE

What if the NFL was 100% luck? By that I mean, "what if the winner of each game was determined as if it were a flip of a fair coin?" The binomial distribution gives us the answer. The distribution mimics a bell-curve normal distribution. The graph below is a histogram of season win totals in a pure luck league.



As we'd expect, it illustrates that in such a league with 16 games, 8 wins would be the most common season outcome. About 20% of all teams would finish 8-8. About 5% of all teams would finish 11-5 and another 5% would finish 5-11. Almost no teams would finish undefeated or winless (each having a 0.00002 probability).

This type of league represents perfect parity. Every team has exactly a 50% chance of winning each game. To spectators (and NFL analysts) however, it would still appear that some teams are "better" than others. Some teams would even appear "hot" because they won several games in row, when in reality it's just an artifact of luck. (Sometimes when you flip a coin you get heads a few times in a row.) Does the coin have momentum? Is it hot? Some coins would have an above average number of heads several seasons in a row. Is that coin a dynasty?

But the real question is: How does the actual distribution of NFL regular season wins compare to the hypothetical luck league? How different is the observed distribution from an idealized distribution of pure luck? The histogram below shows the distribution of the actual NFL regular season win totals for every team since 2002, when the current division structure and scheduling system began. It's slightly irregular because it represents just five seasons (160 team records).



9-7 turns out to be the most common W-L record, followed by 10-6. I didn't expect that. At first, I thought I had discovered something interesting in the "dip" that the distribution takes at 7 wins. I thought that it was evidence that, even more often than we'd expect, teams with playoff hopes usually beat teams with nothing to gain at the end of the season. This effect would result in extra occurrences of 10-game winners. But after running many simulations of random sets of five seasons, irregularities like that were very common by chance alone. (More on that later.)

Let's compare the two distributions--pure luck vs. actual. The next histogram shows both distributions together, and at the same relative scale.



So how different are the distributions? Statistically, they are absolutely not similar. The goodness-of-fit test for two distributions is the chi-square test. It tells us it is infinitesimally unlikely that the actual distribution is sampled from the binomial distribution (p=8.9E-34). But that is obvious enough by just looking at them. To me, it looks like the actual distribution is a flattened version of the binomial distribution. It's as if something is "squashing" the luck distribution to create the actual distribution.

By comparing the two distributions, we can calculate that of the 160 season outcomes, only 78 of them differ from what we'd expect from a pure luck distribution. That's only 48%, which would suggest that in 52% of NFL games, luck is the deciding factor!

To me, that was too hard to accept. Frankly, I didn't buy it, so I kept at it. In part 2 of this article I'll re-attack the question from the opposite direction. I'll compare a theoretical "pure skill" league with the actual NFL win distributions. We'll see that it's skill that's "squashing" the luck into the actual distribution.

Surprise Super Bowl Teams

I thought it would be interesting to take a look at surprise Super Bowl teams in recent years. I wanted to know if the surprise teams could be identified based on their prior performance. Ultimately, I thought I might be able to predict which teams might surprise us in 2007.

With the predicition model's ability to isolate the effect of luck on team wins from performance, it can calculate "expected wins" which tend to be a more accurate reflection of a team's talent and skill than their actual record.

My theory was that potential surprise teams can be identified by looking at teams with mediocre actual W-L records but had notably better expected records in the season prior to their Super Bowl appearance.

The teams I considered as recent surprise teams are listed below, along with their previous season's win totals. If I've overlooked any, I apologize.









































Team YearPrior Wins
NE '966
ATL '987
TEN '998
STL '994
BAL '008
NE '015
CAR '037
SEA '059

I considered the '05 Seahawks as a surprise team despite their division championship the year prior because of the weakness of the division and that they won it with only 9 wins.

Going back to 1995 cost some detail in the data. Compared to my latest model, I could not use fumbles, sacks, or penalties in the regression for estimating season win totals. The model includes offensive and defensive: pass efficiency, interception rate, and run efficiency. The coefficients are derived from all 32 teams from the past five NFL regular seasons (n=160).

The result is a slight drop in model precision. R-squared drops from about 0.70 to 0.62. But for the purposes of this effort, I believe that's acceptable.

If my theory is correct, we would expect to see expected wins for the surprise teams exceed their actual wins by a notable amount. Here is the data table and the calculated expected wins.







































































































TeamO PASSO INT RATEO RUND PASSD INT RATED RUNAct. WinsExp. Wins
NE '955.520.0233.947.480.0274.1964.3
ATL '977.120.0233.727.650.0364.0777.6
TEN '986.710.0194.267.210.0233.8987.7
STL '986.080.0323.516.690.0344.2846.2
BAL '996.150.0374.075.480.0353.1489.9
NE '006.130.0273.286.790.0183.7055.6
CAR '026.420.0473.516.380.0333.6976.6
SEA '046.970.0344.486.820.0414.4998.9


We find there is no tendency for surprise teams to have higher expected wins than actual wins. The only team that shows such a pattern is the '99 Ravens who played well enough to expect 10 wins while only notching 8. We see that the '98 Rams were not quite a 4-win team, and the '96 Patriots were a little more surprising than their 6 wins in '95 indicated. Otherwise, we can't use prior expected wins to identify the Cinderella story of 2007.

Jeff Garcia in Tampa

The Buccaneers have been on a slide since their Super Bowl win in 2002. Although their actual records since then seem slightly erratic, their expected wins based on their internal efficiency stats tell a different story. They have been in consistent decline since the end of the 2002 season.


Although Bruce Gradkowski did well enough for a young back-up QB thrust into the starting roll on a losing team, he still performed well below average ranking 33 of 45 QBs. With the addition of Jeff Garcia, Tampa Bay can expect a boost assuming Garcia can perform as well as he did in Philadelphia in 2006.

Gradkowski performed well enough to contribute -0.70 QB wins added in 2006, while Garcia contributed +1.83 QB wins added.






















































PlayerCmp% RankInt

Rank
Sk RankWins AddedTotal

Rank
Garcia1525+2.33
Gradkowski401723-0.233


Prior to the change in QB, the TB win projection stood at approximately 5 games for 2007. Doing some simple math, we can project TB is expected to win:

5 -(-0.2) + (+2.3) = 7.5 wins

I admit that this projection is based on some optimistic notions. It assumes Garcia stays healthy and plays as well for 16 games in TB as he did over a short 6-game stint on a talented Eagle team. Then again, he could play even better. The bottom line is: look for TB to improve significantly, from a 5 win team to about a 7 win team, mostly on the basis of their upgrade at QB.

A Better QB Passer Rating

The QB passer rating system is roundly criticized for various reasons, yet it continues to be used in the media and by analysts around the NFL. The formula used to create the passer rating is a bizarre, complicated, redundant, incomplete, and arbitrary equation. And it should be replaced.

CURRENT NFL PASSER RATING CRITICISMS

It is bizarre because the result, the actual rating, signifies nothing. The units are in terms of...well nothing--not yards, or points, or completions, or their equivalents.

It is complicated because it combines four separate components that each practically require a slide ruler to compute. For example, the interception component is 2.375-(Int/Att x 25).

It is incomplete because it does not consider sacks.

It is redundant because it includes both completion percentage and yards per attempt. The yards per attempt stat is highly dependent on on completion percentage. Yards per attempt is actually just (yards per catch) x (completion %). Therefore, completion percentage is double counted in the formula.

It is arbitrary because each of the four components are not weighted in any meaningful way. The components of the formula are based on multipliers and constants selected to give the rating a nice scale rather than based on their importance to scoring or winning. They also use arbitrary maximums--each component is capped at 2.375 for an unknown reason. Look at the interception component formula above. Why is it multiplied by 25? Why not 30 or 20?

My personal criticism of the passer rating is that it includes touchdown passes. They should not be included in a passer rating because they are the result of many other factors beyond QB passing proficiency. For example, a QB could be on a team with a terrific defense that frequently produces turnovers deep in an opponent's territory. Or he could benefit from a great running offense that sustains drives. Further, TD passes are the culmination of all the other attributes of the passer. Accuracy, avoiding interceptions, and avoiding sacks all lead to TD passes.

I contend that passer attributes that enable the TD passes are far truer measures of passing performance than whether a WR broke a 7 yard slant pass for 50 yard TD run. Why should the QB get credit for that? What about a perfectly thrown 40 yard pass to a WR knocked out of bounds at the 1 yard line, followed by a 1-yd TD run? We don't need a fantasy football scoring formula, we need a reliable passer rating system that tells us how much a quarterback contributes to his team's success.

BUILDING A BETTER RATING SYSTEM

My goal is to create a meaningful passer rating based on factors controlled primarily by the quarterback alone. The weights of each component are based on their statistically predictive power of team wins. The units of the rating are in "wins added." In other words, the result of the new rating reveals, holding all other team performance measures equal, how many wins above (or below) average, a QB's passing performance would add to his team over the course of 16 games.

The formula is based on a multivariate regression model of team wins. Using data from the past five NFL regular seasons, the regression model estimates team wins based on the efficiency stats of each team including passing, running, turnovers, and penalties. Regression models can hold all other factors equal. By only adjusting the factors of interest (passing performance) we can see the effect on the estimate of season wins. The regression model automatically weights each factor corresponding to its actual importance in terms of team wins. Arbitrary weighting is not necessary.

The "wins added" model of rating athletic performance is based on a rich foundation of previous work in other sports. In baseball, Bill James's runs created statistic weights its various components in a way that approximates a batter's share of runs scored by his team. The recent book Wages of Wins proposed a similar stat for NBA basketball called Wins per 48 Minutes (WP48). The author, David Berri, weighted stats such as points scored, steals, turnovers, and assists based on a regression model of team wins. WP48 estimates each player's share of his team's wins. [I should also note Berri also attempted something similar with QB ratings, called QB Score, with far less success. It weights the components based on a regression and includes sack data. But it contains the same comlexities, redundancies, and other problems with traditional passer rating. In fact, it produces nearly identical results with the traditional rating, correlating at 0.95.]

The new rating system is also a derivative of a baseball stat called the Defense Independent Pitching Statistic (DIPS) developed by amateur sabermetrician Voros McCracken, now employed by the Red Sox. DIPS is a revolutionary concept because it intentionally ignores a large part of conventional pitching statistics. It is based on the idea that once a batter puts a ball into play, the pitcher has no control over whether it is a hit or an out. It is mostly due to luck, defensive positioning, and fielding skills. DIPS considers only those stats that a pitcher has total control over such as strike outs, walks, and home runs. Although DIPS only includes a subset of pitching performance, it is actually more predictive of a pitcher's year-to-year ERA than his actual ERA. It's better because it's a truer measure of pitching ability that ignores the noise of luck and other factors beyond the pitcher's control. The stats that remain are good proxy variables for the ones not included.

This new passer rating system is based on the same concepts as runs created, WP48, and DIPS. Like runs created and WP48, the new passer rating system is computed in terms of a concrete outcome. In this case, it is in terms of team wins instead of team scoring. And similar to DIPS, the new system includes performance measures primarily controlled by the quarterback. Although it is much harder in football than baseball to extricate one player's contribution from the rest of his team, the new system ignores spurious stats such as TD passes.

In summary, the new passer rating:

1. Is not arbitrary. Each component is weighted exactly as much as their relative importance to winning games. These weights are derived from a regression model using data from all teams since the 2002 expansion.

2. Is computed in terms of team wins. The regression model allows the passer rating model's component weights to translate directly into how many additional wins a QB's performance would yield, on average, over 16 full games.

3. Is not redundant. The components do not double count passing stats.

4. Includes only the passing stats primarily controlled by the QB. Factors such as passing yards after catch are not included. Admittedly, this is an imperfect feature of this rating system, but that is due to the team nature of the sport of football. No method could perfectly separate a single player's contribution from his team's.

The three components of the new system are completion percentage, interception rate, and sack yardage rate. In three prior posts, I discuss why I included each one, how they are calculated, and list how the QBs of 2006 rank for each component. The resulting formula of the new passer rating is:

QB Wins Added = (Comp% * 0.18) - (Int/Att * 50.5) - (Sack Yds/Att * 1.57) - 8

Why subtract 8? Because that is the average number of wins achieved by the average QB, and we're interested in 'wins added' above the average performance.

2006 PASSER RATINGS

Now let's look at how the QBs of 2006 fared in the new passer rating system. The total 'wins added' for each quarterback is listed. The 'wins added' for each component of the rating are also listed, along with how each QB ranks among his peers. The traditional NFL passer rating (QBR) is also listed for comparison.


RankPlayer+Cmp% WinsCmp% Rank+Int WinsInt Rank+Sk WinsSk Rank+Total WinsQBR
1Manning1.2530.9550.5012.69101.0
2Brees1.1250.7790.4542.3496.2
3Garcia0.65151.2220.4152.2895.8
4Carr1.8410.4215-0.05292.2182.1
5Huard0.47201.5510.09222.1198.0
6Brunell0.76120.9940.20171.9586.5
7McNair0.8980.49140.4631.8482.5
8Bulger0.8791.083-0.17371.7892.9
9Rivers0.65160.7970.27121.7192.0
10Rattay0.42210.7780.4621.6688.2
11Pennington1.1640.14230.21151.5182.6
12Brady0.67140.61110.23131.5087.9
13Romo1.302-0.12320.19191.3795.1
14Palmer0.76130.52120.08251.3693.9
15Warner1.1260.3020-0.16351.2589.3
16Delhomme0.53190.50130.16201.1882.6
17Leftwich0.17260.41160.3470.9279.0
18Johnson0.62170.09250.06270.7672.0
19McNabb-0.19320.8160.08240.7095.5
20Losman0.80100.1622-0.36410.5984.9
21Frye1.127-0.3438-0.21400.5772.2
22Kitna0.7811-0.0430-0.19390.5479.9
23Favre-0.37360.31190.4060.3472.7
24Garrard0.3823-0.06310.02280.3480.5
25Culpepper0.42220.6510-0.78450.2877.0
26Manning-0.07290.07260.20180.2077.0
27Leinart-0.23330.20210.11210.0874.0
28Smith0.0028-0.01270.07260.0674.8
29Harrington-0.1030-0.12330.28100.0568.2
30Cutler0.1825-0.0229-0.16340.0088.5
31Green0.5418-0.4439-0.1636-0.0674.1
32Campbell-0.90430.33180.338-0.2376.5
33Gradkowski-0.73400.41170.0823-0.2465.9
34Roethlisberger0.2924-0.6142-0.1232-0.4375.4
35Plummer-0.5237-0.23350.2114-0.5468.8
36Grossman-0.6339-0.27370.299-0.6073.9
37Hasselbeck-0.2734-0.2134-0.1533-0.6376.0
38Wallace0.0227-0.6343-0.1131-0.7276.2
39Brooks-0.1431-0.2736-0.5244-0.9261.7
40Young-1.1845-0.02280.2016-1.0066.7
41Vick-0.99440.1224-0.3642-1.2375.7
42Bledsoe-0.8641-0.5341-0.1738-1.5669.2
43Simms-0.6138-1.37440.2811-1.7146.3
44Anderson-0.3035-1.4845-0.0930-1.8763.1
45Walter-0.8642-0.5240-0.5143-1.8955.8

We see that Manning, Brees, and Garcia top the list, but some surprises follow. QBs such as David Carr, Mark Brunell, and Tim Rattay actually fare pretty well. Each of those three QBs place in the top 10 for different reasons, as you can see by how they rank in each component. Those QBs performed fairly well despite being surrounded by below-average teams.

McNair ranks high at number 7. Although he wasn't starting on anyone's fantasy team last year (16 TDs all year), he led Baltimore to 13 wins by avoiding sacks and keeping the ball out of the hands of the opponent's defense.

One final note: I think this formula is incomplete. You might notice that QBs who tend to throw short passes tend to rank highly. These QBs tend to have good completion % and interception stats. The formula should include this consideration. I think the best way would be to use yards per completion minus yards after catch. The stat would be called "air yards" or something catchy. Unfortunately, YAC totals for quarterbacks are not publicly available--I've searched everywhere. I know it is kept by Stats, Inc., but I can't find it for any year after 2002. If anyone knows where to find QB YAC stats, please leave a comment.

QB Wins Added 3

Recently, I began constructing a passer rating system that, to the greatest degree possible, includes performance variables controlled by the quarterback alone and avoids redundant use of variables that measure the same attribute. This improved system includes completion percentage, interception rate, and sack yards per attempt. The previous two posts discussed completion percentage and interception rate. This post will examine sacks. Each of the stats used in the rating system are computed in terms of how many wins they add, on average, to a QB's team over the course of a season. The wins added are based on the previously established regression model based on team efficieny stats.

More and more, I'm coming to realize that sacks are far more dependent on the quarterback than on his pass protection. A quarterback's mobility, release, decision-making, and pocket presence all are critically important. I'm using sack yardage rate instead of total absolute sacks for two reasons. First, the number of pass attempts must be accounted for. We would expect a QB with more pass attempts to be sacked more often than a QB who passes less often, given equal abilities to prevent sacks. Second, sack yards yield more information about the QB than sacks alone. Some QBs have fewer sacks, but because they often gamble by scrambling deep in the backfield, they give up large numbers of sack yards. Does the QB tend to step up in the pocket or drift backward? As you'll see, some of the QBs considered the most mobile actually give up some of the most sack yards.

The table below lists each quarterback with at least 100 pass attempts in 2006, their sack yards, and their sack yardage rates (sack yards per pass attempt). The column labeled "Sk Yd Rate > Avg" is the sack yardage rate above (or below) league average. The next column, "+Sk Yd Wins" lists the wins added based on sack yards for each quarterback. The list is sorted in order of best to worst.













































































































































































































































































































































PlayerSk YdsSk Yd

Rate
SkYdRate > Avg+Sk Wins
Peyton Manning860.15-0.280.44
Tim Rattay180.17-0.260.41
Steve McNair840.17-0.260.41
Drew Brees1050.18-0.250.39
Jeff Garcia400.21-0.230.36
Brett Favre1340.21-0.220.35
Byron Leftwich480.25-0.180.29
Jason Campbell550.26-0.180.28
Rex Grossman1420.28-0.150.24
Joey Harrington1160.29-0.150.23
Chris Simms320.29-0.140.22
Philip Rivers1440.30-0.140.22
Tom Brady1750.32-0.110.17
Jake Plummer1110.33-0.100.16
Chad Pennington1720.33-0.100.16
Vince Young1290.34-0.100.15
Mark Brunell920.34-0.100.15
Eli Manning1860.34-0.090.15
Tony Romo1240.35-0.090.14
Jake Delhomme1670.37-0.070.10
Matt Leinart1580.40-0.040.06
Damon Huard1060.41-0.030.04
Bruce Gradkowski1460.41-0.020.03
Donovan McNabb1400.42-0.020.03
Carson Palmer2330.42-0.010.02
Alex Smith2020.42-0.010.02
Brad Johnson2000.43-0.010.01
David Garrard1190.460.02-0.03
David Carr2400.500.06-0.10
Derek Anderson660.530.09-0.15
Seneca Wallace830.540.10-0.16
Ben Roethlisberger2800.540.11-0.17
Matt Hasselbeck2290.570.13-0.21
Jay Cutler850.570.13-0.21
Kurt Warner1040.570.14-0.22
Trent Green1270.570.14-0.22
Marc Bulger3660.570.14-0.22
Drew Bledsoe1070.580.14-0.23
Jon Kitna3880.590.15-0.24
Charlie Frye2620.600.17-0.26
J.P. Losman3320.700.26-0.41
Michael Vick3030.700.27-0.42
Andrew Walter2560.800.36-0.57
Aaron Brooks1740.800.36-0.57
Daunte Culpepper1500.970.53-0.84
Average155.90.43

QB Wins Added 2

In the last post, I began constructing a passer rating system that, to the greatest degree possible, includes performance variables controlled by the quarterback alone and avoids redundant use of variables that measure the same attribute.

The new system includes completion percentage, interception rate, and sack yards per attempt. The previous post discussed completion percentage, and this one will cover interception rate. Each QB's stats are computed in terms of how many wins they add, on average, to his team over the course of a season.

While most fans and analysts consider how many inteptions a QB has thrown, they ignore the fact that various QBs have a wide range of passing attempts, even among QBs with the same number of games played. Interceptions per pass attempt is a better indicator of a QB's true tendency to throw interceptions.

The table below ranks QB's interception rates in 2006. Only QBs with over 100 pass attempts are included. Theh column labled "Int Rate > Avg" is the amount above (or below) the average NFL interception rate. Negative numbers are better here because they mean fewer interceptions per attempt. The column labled "Int Rate Wins" are the wins added (or lost) due to the interception rate of the listed QB. Postive numbers are (obviously) better here. The added wins are computed by the efficiency win model Ipreviously established.



















































PlayerInt RateInt Rate >

Avg
Int Rate Wins
Damon Huard0.004-0.0271.385
Jeff Garcia0.011-0.0211.060
Marc Bulger0.013-0.0180.913
Mark Brunell0.015-0.0160.826
Peyton Manning0.016-0.0160.788
Donovan McNabb0.019-0.0130.650
Philip Rivers0.019-0.0120.622
Tim Rattay0.019-0.0120.611
Drew Brees0.019-0.0120.608
Daunte Culpepper0.022-0.0100.485
Tom Brady0.023-0.0090.443
Carson Palmer0.024-0.0070.360
Jake Delhomme0.025-0.0070.334
Steve McNair0.025-0.0070.329
David Carr0.026-0.0050.256
Byron Leftwich0.027-0.0050.248
Bruce Gradkowski0.027-0.0050.243
Jason Campbell0.028-0.0030.169
Brett Favre0.029-0.0030.151
Kurt Warner0.029-0.0030.132
Matt Leinart0.031-0.0010.033
J.P. Losman0.0320.000-0.005
Chad Pennington0.0320.000-0.022
Michael Vick0.0320.001-0.046
Brad Johnson0.0330.002-0.077
Eli Manning0.0330.002-0.092
Alex Smith0.0350.003-0.173
Vince Young0.0350.004-0.183
Jay Cutler0.0350.004-0.187
Jon Kitna0.0360.004-0.207
David Garrard0.0360.004-0.227
Tony Romo0.0370.006-0.284
Joey Harrington0.0370.006-0.288
Matt Hasselbeck0.0390.007-0.371
Jake Plummer0.0390.008-0.398
Rex Grossman0.0400.008-0.429
Aaron Brooks0.0400.008-0.429
Charlie Frye0.0420.010-0.508
Trent Green0.0430.012-0.604
Andrew Walter0.0450.013-0.680
Drew Bledsoe0.0450.014-0.691
Ben Roethlisberger0.0470.015-0.770
Seneca Wallace0.0470.016-0.797
Chris Simms0.0620.030-1.537
Derek Anderson0.0640.032-1.641
Average0.032



In the next post, I'll look at sacks. Then finally, I'll pull it all together for a comprehensive QB win-rating.

QB Wins Added 1

In this series of posts I'm going to build a better quarterback rating system. The system will be try to isolate the passing stats that the QB alone is primarily accountable for. The stats will be efficiency stats in that they are in terms of attempts. The system will also attempt to exclude redundancy stats.

The official Passer Rating system includes co-dependent factors such as TDs, interceptions, and completion percentage. But an accurate QB will tend to have both a high completion rate and high TDs. The passer rating, therefore, double counts accuracy. Further, a QB's passing TDs may be due to his defense's ability to get the ball turned over to him in his oppenents' territory, or due to an effective running game, as much as due to his own ability. The goal is to isolate what the QB is truly responsible for and filter out what the QB does not control.

The system will include completion percentage, interceptions per attempt, and sack yards per pass attempt. Although interceptions and sacks are not solely attributable to the QB, the stats imply that the QB is overwhelmingly responsible for them. Take the case of McNair in Baltimore. After replacing Kyle Boller, Baltimore went from one of the league's worst sack rates to one of the best. The rest of the offense, including the offensive line, was completely identical.

Each stat in the system will be translated into team wins by the efficiency win model previously established. In the first post we'll rank quarterbacks in terms of their 2006 regulare season completion percentage. We'll compute how many percentage points above or below average each QB was last year. All other factors being equal, every additional point of completion percentage brings an extra 0.18 wins to a team on average. Therefore we can compute how many wins each QB contributed to his team by his completion percentage alone.

(Only QBs with more than 100 attempts are considered.)


















































PlayerComp PctPct > AvgWins Added
David Carr68.38.891.60
Tony Romo65.35.891.06
Peyton Manning65.05.591.01
Chad Pennington64.55.090.92
Charlie Frye64.34.890.88
Kurt Warner64.34.890.88
Drew Brees64.34.890.88
Steve McNair63.03.590.65
Marc Bulger62.93.490.63
J.P. Losman62.53.090.56
Jon Kitna62.42.990.54
Carson Palmer62.32.890.52
Mark Brunell62.32.890.52
Tom Brady61.82.390.43
Philip Rivers61.72.290.41
Jeff Garcia61.72.290.41
Brad Johnson61.52.090.38
Trent Green61.11.690.30
Jake Delhomme61.01.590.29
Damon Huard60.71.290.23
Daunte Culpepper60.40.990.18
Tim Rattay60.40.990.18
David Garrard60.20.790.14
Ben Roethlisberger59.70.290.05
Jay Cutler59.1-0.31-0.06
Byron Leftwich59.0-0.41-0.07
Seneca Wallace58.2-1.21-0.22
Alex Smith58.1-1.31-0.24
Eli Manning57.7-1.71-0.31
Joey Harrington57.5-1.91-0.34
Aaron Brooks57.3-2.11-0.38
Donovan McNabb57.0-2.41-0.43
Matt Leinart56.8-2.61-0.47
Matt Hasselbeck56.6-2.81-0.51
Derek Anderson56.4-3.01-0.54
Brett Favre56.0-3.41-0.61
Jake Plummer55.2-4.21-0.76
Chris Simms54.7-4.71-0.85
Rex Grossman54.6-4.81-0.87
Bruce Gradkowski54.0-5.41-0.97
Andrew Walter53.3-6.11-1.10
Drew Bledsoe53.3-6.11-1.10
Jason Campbell53.1-6.31-1.14
Michael Vick52.6-6.81-1.23
Vince Young51.5-7.91-1.42
Average59.4

In the next post, I examine interception rate. Then in the third part of this this article, I'll look at sack yardage rate. Finally, I'll sum the wins added for each QB to produce a more valid passer rating system.

How Many Wins is Manning Worth?

In a recent comment at another site, someone mentioned that "Peyton Manning was good for at least 10 wins a year." I thought that sounded about right.

Using the QB rule-of-thumb I made the other day, I'll estimate how many wins Manning is good for on the basis of his accuracy alone. By accuracy I mean his completion percentage and interceptions.

Manning's completion percentage over the last 4 years hovers around 65.0% while the league average is 59.4%. His interceptions average at 9.8 per year while the NFL average is 16.4.

Based on the coefficients from the efficiency-win model, we can do some quick math.

Cmp%: 65.0 - 59.4 = 5.6% * 0.18 wins/cmp% point = 1.32 wins
Int: 16.4 - 9.8 = 6.6 Interceptions * 0.11 wins/int = 0.62 wins
1.32 + 0.62 = +1.94 wins

So about 10 wins, compared to league, average is right. There are also other unquantifiable factors, such has his leadership or audible calling that probably contribute to his success. But all other things being equal, Peyton Manning is worth (at least) 10 wins, right out of the gate.

McGahee in Baltimore

Jamal Lewis has been on a slide since his monster year in 2003 when he threatened Dickerson’s single season rushing record. Since then Lewis has declined from 5.3 yds per rush to 3.6. Even his receiving has declined steadily, except for 2004 when his average was distorted by a very small number of catches (10).

Year  G  Att  Yds   Avg  Rec Yds   YPC
----------------------------------------
2000 16 309 1364 4.4 27 296 11.0
2002 16 308 1327 4.3 47 442 9.4
2003 16 387 2066 5.3 26 205 7.9
2004 12 235 1006 4.3 10 116 11.6
2005 15 269 906 3.4 32 191 6.0
2006 16 314 1132 3.6 18 115 6.4


Willis McGahee’s performance in Buffalo over the past three seasons has been solid but unspectacular, with an average of 3.9 yds per rush.

Year  G   Att   Yds  Avg  Rec Yds  YPC
----------------------------------------
2004 16 284 1128 4.0 22 169 7.7
2005 16 325 1247 3.8 28 178 6.4
2006 14 259 990 3.8 18 156 8.7

Assuming McGahee gets the same number of carries and receptions as Lewis did in 2006, we can estimate the Ravens' new team rushing and passing efficiency. Although that assumption is unlikely because Baltimore will likely use spread sets, it gives us a feel for the difference between the two running backs. McGahee gives Baltimore 0.3 extra yds per rush and 1.2 extra yds per catch.

314 carries x 0.3 yds/rush = 51.4 added rushing yards
18 catches x 1.2 yds/catch = 21.6 added passing yards

+51.4 yds / 476 team rush attempts = +0.11 yds/rush team rushing efficiency
+21.6 yds / 525 team pass attempts = +0.04 yds/att team passing efficiency

If we run the new efficiencies through the efficiency win model, we see that McGahee could be worth half a win.


I realize it’s a little abstract to say a player can add 0.42 wins to a team’s record. What the math says is: given a similar number of carries and receptions, and given McGahee’s slightly better stats, the Ravens could expect to average 0.42 more wins if the season were played out a million times.

Compared to what we saw with quarterbacks McNair and Brees, who were each estimated to bring an extra 2 or more wins to their new teams, 0.42 added wins seems relatively modest. For a running back to make a very large impact on a team’s record, he would have to be a 5 yd/carry type of back. LaDanian Tomlinson, for example, averaged 5.2 yds/rush in 2006.

Personally, I think these estimates are conservative. Lewis’s strength was in the straight-ahead runs, and not much else. From all the reports, Baltimore intends to use McGahee in a variety of formations making their offensive more versatile and less predictable.

Defense Penalties Puzzle Solved

In an earlier post, I found that penalties committed by a team's defense correlated postively with team wins. In other words, the more penalties committed on defense, the more games a team could expect to win. I was confused by this result and re-checked my math several times, but it always came out the same.

I finally realized the cause of the result. What the stat services call defensive penalties are actually penalties committed by opposing teams on offense and defense. And what they call offensive penalties are penalties committed by one's own team on offense and defense. All the stat sites that I've used for data use the same nomenclature: nfl.com, sportsline.com, and espn.com. I don't know why it's done like that.

It seems all the big websites use data from Stats, Inc., so I bet that's how they do it there. Unfortunately, NFL gamebooks, the official box scores of NFL games, do not breakout a team's penalties between offense and defense.

As an explanatory variable, it's valid--opposition penalties help explain what makes teams win. But as a predictive variable for season wins, it's useless and should be removed. This means some revisions are in order for my models. The good news is that it doesn't change the models' predictions in a significant way because the solution is to remove the variable. And fortunately, a team's opposition's penalty stats tends to average out throughout the season, so removing a variable that is very average doesn't effect the dependent variable (team wins) appreciably.

Trent Green in Miami

How many added wins can Trent Green bring to Miami? First, let’s look at his recent performance in KC. I'm going to include his most recent year despite his head injury early in the season because I believe it's representative of what the Dolphins may expect from Green. His average stats for the last four seasons are:


         CMP%      YPA     YD/CMP    INTRATE    SKRATE
-------------------------------------------------------
GREEN 63.3 7.7 12.1 0.027 0.43

Green’s projected stats and, for comparison, Miami’s 2006 passing stats are:


         CMP%      YPA     YD/CMP    INTRATE    SKRATE
-------------------------------------------------------
PROJ 63.3 7.0 11.1 0.027 0.34
MIA 06 58.0 6.1 10.5 0.032 0.27
(Interception rate is defined as interceptions per pass attempt. Sack rate is defined as sack yards lost per pass attempt. )

Green’s biggest problem in KC last year was avoiding the sack. After returning from his head injury, he was sacked on 12% of his pass plays, more than double his rate in any other year. If the Dolphin’s line can protect Green, he should be a solid upgrade to a woeful passing attack.

All other factors being equal, we can expect Trent Green to bring about +2.0 added wins to Miami. Assuming the other parts of the Dolphins team perform as well as last year, they would likely end the season with 10 wins.

New QB Added Wins

In this article, I’ll explain how I estimated how many additional wins an established quarterback can bring to new team by forecasting his likely effect on team passing stats.

First, we need an idea of which stats the quarterback on an NFL team is primarily responsible for. Team passing performance is dependent on every member of the offense, so it is very difficult to parse what can attributed to the quarterback and what can be attributed to other players. We’ll need to make some assumptions from a qualitative look at the NFL records.

These assumptions are only rough estimations of the stats we need to estimate wins. At this point, it would be impossible to declare exactly how much responsibility a QB has for certain team stats, and even if we could it probably varies from team to team. I’m only going to use them to get a back-of-the-envelope estimation for the effect of acquiring a veteran QB. Assuming all other things stay the same, would he add 5 wins, 2 wins, or none?

PROJECTIONS

Completion percentage appears to be entirely owned by the QB. Year-to-year deviations in completion % tend to be very narrow. QB careers are typically marked by one low percentage rookie year, followed by a steady completion rate in subsequent years. Team changes do not affect completion % any more than the small year-to-year variations while playing for the same team. We’ll say that the QB owns completion percentage.

Yards per catch varies significantly from year-to-year. It seems to be far more dependent on the QB’s supporting cast than completion %. My rough estimate is that the QB is responsible for 1/3 of the yards per catch stat. The other 2/3 is most likely due to receiver performance, and yards-after-catch in particular. Another factor might be the type of offensive scheme used by his offense. We’ll say that the QB owns 1/3 of yards per catch and the other 2/3 is owned by the team’s receivers or other factors.

(Yards per catch)*(completion %) = yards per attempt, the primary pass stat used in the win estimation model. So we can already approximate it.

We also want to consider sacks. Sack yards are generally due to a combination of three things: QB immobility/poor decision, pass protection failure, or a failure of the receivers to get open/run the correct route. QB sacks do appear to vary significantly from year-to-year, suggesting a quarterback’s share of responsibility for his sacks is no more than 50%. We’ll say that 1/2 of a QB’s sack rate is owned by the QB himself.

The last stat we need to estimate is interception rate. Total interceptions vary from year to year, but interceptions per pass attempt is much steadier. And although the cause of an interception can be contributed to by a receiver breaking the wrong way or a QB being hurried by a pass rusher, it is the QB’s ultimate decision to throw the ball. There are also freak interceptions, such as bobbled passes, but we’ll assume those types of plays tend to even out among QBs over time. Ultimately, the decision to throw and the location of the pass rests on his ability alone. We’ll say the QB completely owns his interception rate.

We can use two recent cases of veteran QBs coming to new teams to see if the assumptions and projections make sense. Both Drew Brees and Steve McNair had significant impacts on their new teams.

DREW BREES IN NEW ORLEANS

Averaging Brees’s last 4 (healthy) seasons at San Diego as a baseline, his typical performance yields the following stats. (Notes: YPA does not yet factor in sack data. Sack rate is defined as sack yards per pass attempt.)

       CMP%      YPA     YD/CMP    INTRATE    SKRATE
-----------------------------------------------------
BREES 62.6 7.0 11.1 0.030 0.40

New Orleans’s 2005 passing stats were:

       CMP%      YPA     YD/CMP    INTRATE    SKRATE
-----------------------------------------------------
NO 05 55.6 6.5 11.6 0.043 0.47

Replacing NO’s 2005 passing stats with the forecast stats due to Brees arrival we get the following projection. Below the projection are the actual 2006 stats for NO.

       CMP%      YPA     YD/CMP    INTRATE    SKRATE
-----------------------------------------------------
PROJ 62.6 7.2 11.6 0.030 0.44
NO 06 63.0 8.0 12.4 0.020 0.22

Brees had a career year, and our projections are sure to undervalue his contribution to NO’s improved record. By using a win model based on efficiency, we can compare the projected wins “added” by Brees and the actual wins added. (Sack data is now factored in to pass efficiency.)



How to read the chart: The first data column lists the actual efficiency stats of NO in 2005. The expected wins given their stats in ’05 was 4.7 wins, although they only won 3. The second column lists the efficiency stats we would project with the assumptions discussed above. The third column lists the actual stats for NO in ’06. The next column, titled +QB wins, indicates how many added wins our projections would attribute to the arrival of Drew Brees in NO. The final column lists the actual added wins attributed to the actual performance of the entire Saints team in 2006.

In the case of Brees, the assumptions would have underestimated his impact. Brees had a career year in ’06. The projection would have estimated +2.0 added wins attributed to him. In actuality, his contribution was almost double the projection at +3.9 added wins. Other team improvements added another +2.7 wins.

I think the primary reason for the disparity is the arrival of receivers Hopper and Colston, and an amazing 23.3 yards/catch from 2nd year receiver Henderson. I’m sure Brees had a lot to do with their success, but his new receivers’ success would have been impossible to predict. At the very least, our projections can say how many wins a quarterback could add to a new team, given that he has a career year--a best case scenario.

STEVE MCNAIR IN BALTIMORE

Averaging McNair’s last four (healthy) seasons in Tennessee, we get this baseline:


         CMP%      YPA     YD/CMP    INTRATE    SKRATE
-------------------------------------------------------
MCNAIR 61.6 7.1 11.6 0.028 0.33

Baltimore’s passing stats in 2005, split between Kyle Boller and Anthony Wright, were:


         CMP%      YPA     YD/CMP    INTRATE    SKRATE
-------------------------------------------------------
BAL 05 59.4 6.0 10.1 0.037 0.49

Replacing BAL’s passing stats with the projected stats due to McNair’s arrival, and the actual stats from 2006 are listed below.


         CMP%      YPA     YD/CMP    INTRATE    SKRATE
-------------------------------------------------------
PROJ 61.6 6.5 10.6 0.028 0.33
BAL 06 63.0 6.5 11.3 0.025 0.21

McNair was kind enough to match our projections much better than Brees. Unlike Brees, McNair was working with a receiving corps in Baltimore that was unchanged from ’05 to ’06. The biggest difference between the projections and the actual stats was in the sack rates.

We would have predicted that McNair would bring +1.8 added wins to the Ravens in ’06. He actually performed well enough to contribute +2.3 added wins, but “about 2 wins” is good enough for the back of the envelope. The biggest difference for the Ravens between ’05 and ’06 was in the defense’s interception rate which added +2.7 wins to the bottom line.

Although the projections are well in line with the actual results in ’06, we still may be discounting McNair’s presence. The difference between the 10.2 projected wins and the 13 actual wins might be due to his three game-winning 4th quarter drives against SD, CLE, and TEN. All three games appeared out of reach to Baltimore fans who were accustomed to the pre-McNair passing offense.

RULES OF THUMB

When I was a fighter pilot, I loved rules of thumb. At the speed of sound I didn't have time for a lot of math, so quick, simple, but accurate estimates helped prevent me from running out of gas. Here are a couple quick and simple rules for estimating a quarterback's statistical impact on his team's wins.

Completion percentage directly affects our model's most important statistic, pass yards per attempt. Using the league average yard per completion average of 11.5, every point of completion percentage adds 0.115 yds/att. (11.5 * 0.01 = 0.115). Every 0.115 yds/att added to pass efficiency adds 0.18 expected wins. About a 5 point increase would be needed for an extra win based on completion percentage alone.

Interceptions are also critical. Based on the league average of 519 pass attempts, every interception increases the interception rate by 1/519 = 0.18%. In turn, every 0.18% increase in interception rate reduces expected team wins by 0.11 wins. About every 10 interceptions throughout the season results in 1 fewer expected win.

In the next post I will apply the projection model onto the Miami Dolphins' new quarterback, Trent Green.

What Makes Teams Win? 4

This is the 4th and final part of an article discussing the relative importance of factors in winning NFL games. Part 1 is here, part 2 is here, and part 3 is here.

ANALYSIS

A good passing game is far more important than a good running game in the NFL. It’s at least twice as important, and probably even more so. If we include interceptions as part of the passing game, passing efficiency and interception rates dwarf the importance of running efficiency by a factor of 4 to 1.

An alternate way of looking at interceptions is that they are a threat to the passing game, so their importance should be subtracted, not added, to passing efficiency to properly compare running and passing. Although it's a valid consideration, you cannot win without passing, and so the risk of interception is not optional. The bottom line is that if a team would rather be good at passing or good at running, it should choose passing.

Part of the conventional wisdom about the running game in the NFL is that it “sets up” the passing game. It keeps the defense off balance and unable to focus exclusively on defending against the pass. This may be true up to a point, but it appears that the balance between defending against the run and the pass is far out of equilibrium.

If a good running game gives a team an advantage in passing then we would see a positive and significant correlation between offensive running and passing efficiency. In fact, the correlation is 0.13, which is very weak and not statistically significant. Running well does not prevent interceptions either. The correlation between running efficiency and interception rates is 0.12. Further, including an interaction variable (running * passing) in the regression model results in an insignificant coefficient and a marginally weaker model. We have to conclude that running and passing are fundamentally independent of one another.

It appears that offensive passing is more important than defensive passing, and that offensive running is more important than defensive running. We might conclude that offense is more important than defense, but it may not be that simple. For starters, that violates the symmetry of the sport. Points allowed are equally as important as points scored.

One explanation becomes clear when you add the coefficients of offensive passing and offensive interceptions (1.14 + 0.45 = 1.59). Compare that with the sum of defensive passing and defensive interceptions (0.92 + 0.76 = 1.68). We see that the balance between offensive passing and defensive passing starts to equalize.

Comparing the sum of all offensive weights with the sum of all defensive weights yields a very balanced result. Offensive variables add up to a total weight of 2.38 vs. a total weight of defensive variables of 2.34. Although the defensive variables appear slightly stronger, the relative sums of the weights are within 1.7%--remarkably balanced.

Passing is indeed far more important than running, and although offense appears more important than defense, they're equally important.

What Makes Teams Win? 3

This is the third part of a four-part article discussing the relative importance of factors in winning NFL games. Part 1 is here and Part 2 is here.

CORRELATION SUMMARY

So far we’ve analyzed each phase of the game and its statistical connection with regular season team wins. Below is a table that lists the relevant statistics and their correlations. The table is sorted in order of absolute strength of correlation.


StatWin Correlation
Off Pass Yds/Att0.61
Def Pass Yds/Att-0.47
Off Fumble Rate-0.46
Off Int Rate-0.45
Def FFumble Rate-0.41
Def Int Rate 0.39
Off Pen Rate-0.37
Off Run Yds/Att0.18
Def Run Yds/Att-0.04


The table is presented graphically below. Negative coefficients, such as defense pass efficiency, are shown as positive values to make it easier to compare each variable's relative importance.

The relative importance of each aspect of the game begins to come into focus. Passing is most important, followed by turnovers, then penalties and running. For every aspect, the correlation on the offensive side of the ball is stronger than on the defensive side.

But this isn’t the final word. Correlation coefficients by themselves do not take into account the other factors. In other words, they ignore the effect of the other stats when calculating the correlation.

REGRESSION

To take all facets of the game into account simultaneously and produce a valid model of winning NFL games, we can use linear regression to estimate coefficients for each stat. The relative value of the coefficients will reveal the relative importance of each phase of the game, holding all other variables constant. This will yield estimates that are more pure and accurate than simple correlations.

The dependent variable of the regression model is regular season wins. The independent variables are the efficiency stats I’ve previously outlined. The data set continues to be all 32 teams over the past 5 seasons for a total of 160 observations. The results of the regression are detailed below.



VARIABLECOEFFICIENT
const5.31
O Pass Eff1.43
D Pass Eff-1.65
O Int Rate-53.50
D Int Rate81.70
O Fum Rate-49.10
D FF Rate70.90
O Run Eff1.00
D Run Eff-0.55
Pen Rate-2.73
R-squared0.802


Each of the independent variables are statistically significant at the 0.05 level or better, except defensive run efficiency, which is significant at 0.06. The R-squared value indicates an extremely good overall fit for the model. 80% of the variance in team wins can be explained by the included variables. The remaining 20% could be due to any number of factors, but we have to accept that outcomes in any sport are partly due to luck.

Using the regression results we can estimate a team’s expected wins using a linear equation. Here is what the equation would look like:

Wins = 5.31 + (1.43 * O Pass Eff) + (- 1.65 * D Pass Eff) + …

The regression coefficients are stated in terms of wins per unit of the variable. For example, the coefficient for offensive pass efficiency (yds/att) is 1.43. So for every 1 yard improvement in pass efficiency a team can expect 1.43 additional wins. When coefficients are stated this way, it makes it very easy to estimate the effect on the dependent variable (wins) given a change in one of the independent variables. But it makes it very difficult to get a sense of the relative importance of each variable. Defensive forced fumble rates are certainly not 70 times more important than offensive run efficiency.

To reveal the true relative importance of each factor, we need to standardize each variable by calculating the number of standard deviations from its average value. In statistics, these are known as “normalized” or "standardized" variables, noted by the prefix “z.”

Here are the regression results again, this time calculated with standardized coefficients. The significance of each variable, and the overall fit of the model remain the same since only the units of the variables have changed.



VARIABLECOEFFICIENT
constant8.06
Z O Pass Eff 1.14
Z D Pass Eff-0.92
Z O Int Rate-0.45
Z D Int Rate0.76
Z O Fum Rate-0.33
Z D FF Rate0.42
Z O Run Eff0.46
Z D Run Eff-0.24
Z Pen Rate-0.39


It may seem like we’ve gone through a tortured process to arrive at these coefficients. But they are merely the mathematical weight we would need to give each factor to have the best estimate of actual team wins. These are based on real-world data from every team’s season between 2002 and 2006.

Here is a graph representing each variable’s relative weight. Negative coefficients, are shown as positive values.



Probably the simplest way to interpret the chart is this way. If my team is average in absolutely everything, I'd expect to win 8 games. But if my team is average in everything except offensive pass efficiency, in which we're one standard deviation above average, I'd expect to win 9.14 games (8 + 1*1.14).

So if my team was the league's best at running the ball, say 2.5 standard deviations above average, but average at everything else, we'd expect to win 9.15 games (8 + 2.5*0.46). Compare that to passing--if my team were average at everything but best in the league in passing, we'd expect to win 10.85 games (8+2.5*1.14).

Continue reading the fourth and final part of this article.

What Makes Teams Win? 2

This is a contiuation of an article discussing the relative importance of factors in winning NFL games. Part 1 is here.

TURNOVERS

Turnovers can be analyzed in the same manner. If we want to know how good a defense is at intercepting passes, we should look at interceptions per pass and not total interceptions. Offenses may shy away from passing the ball if a secondary is known to be excellent at generating interceptions. Their ‘total interceptions’ stat would suffer as a result of teams leaning towards the run, but their interception efficiency would tell the real story.

Fumbles are slightly different. Once the ball is on the ground, who recovers it is a very random thing. Fumble recoveries are almost purely the result of unplanned conditions, such as where players happen to be and who happens to be facing the direction of the ball. (Note: Skills like running, passing, or generating interceptions are repeatable and enduring skills which tend to carry over from year to year. Fumble recovery percentage does not.) The best measure of the fumble, therefore, is the forced fumble, which does carry over from year to year. Defenses can be good at jarring the ball loose, but who recovers the ball is random. The more often the forced fumble, the more opportunities the defense has of recovering. Forced fumble rates correlate better with wins, which bolsters its case as the stronger factor in winning.


StatWin Correlation
Def Fumbles0.33
Def Forced Fumbles0.41


On the offensive side of the fumble, the opposite is true. Unforced fumbles are the repeatable, enduring skill--or I should say avoiding the unforced fumble is the skill. Therefore, the fumble per play stat is the optimum stat to measure an offense’s tendency to fumble. Keep in mind that fumbles happen on both pass and run plays.

SPECIAL TEAMS

So far I’ve ignored special teams. There’s a reason for that—they don’t really matter on a consistent basis. Of course, we can remember a critical play here, a miracle play there--they can change the outcome of a game on rare occasions. We can remember these spectacular plays because they are so exceptional. But no matter how we measure special teams in a meaningful way, the correlation coefficients are not statistically significant. That’s not to say that teams can ignore special teams. Every advantage a team can find is important, but in the long run, the importance of kickoffs, punts, return yards, and field goal percentages are all dwarfed in relation to things like running, passing, and turnovers.

There are several valid explanations for the relative unimportance of special teams. First, and perhaps most obvious, is that special teams plays are far less frequent than pass or run plays. Second, the difference between the best special teams and the worst special teams is small in comparison to the spread between the best and worst offenses and defenses. Third, special teams stats are very hard to evaluate. For example, average net punt yards negatively correlate with winning at -0.17. This means the better the team is at punting, the fewer wins it can expect. But I believe this is misleading because teams with good offenses would be able to kick more “inside the 20 yard line” punts than teams that often kick from their own 30 yard line.

One last point about special teams: Big special teams plays, such as kickoff returns for touchdowns, are rare. In the past five seasons, there has been an average of 13.6 kickoffs returned for touchdowns--by all 32 teams combined. The best test of skill vs. luck I have come across asks the question, “does it tend to be repeatable?” Kickoff or punt returns for touchdowns, or 60+ yard field goals, unfortunately, do not tend to be repeatable.

PENALTIES

Penalty rates, however, do tend to be repeatable. Teams that have committed lots of penalties tend to continue to commit more, and vice versa. Experience, discipline, preparation, and playbook complexity probably all contribute to penalty rates. Other factors probably come into play. For example, it might be expected that weak secondaries would tend to commit more pass interference penalties in desperation to prevent deep passes.

Penalties and penalty yards correlate negatively with winning, as we’d expect. (Penalty rate is defined as penalty yards per play.)



StatWin Correlation
Team Penalties-0.21
Team Pen Yds-0.18
Team Pen Rate-0.37


The penalty rate stats have the stronger correlations with wins than either penalties or penalty yards.

Continue reading part 3 of this article.

What Makes Teams Win? 1

This is the first part of a four part article summarizing the relative importance of each phase of the game in winning.

INTRODUCTION

Passing is much more important than running, and offense appears to be more important than defense. Turnovers matter a great deal, interceptions more than fumbles. Penalties matter too, but not like you’d think.

We've been told by NFL analysts for years that defenses win championships and that a solid running game is the key to winning. I'd agree that things appear that way. Teams with lots of passing yards aren't always winners, but teams with lots of running yards almost always are. But, as you're about to find out, appearances can be deceiving.

One way to measure the relative importance of passing vs. running, or offense vs. defense, is to measure their statistical correlations with season wins. For example, the correlation coefficient of a team's total rushing yards correlates with its number of wins in a season is 0.45. A correlation of 1.0 would be perfect correlation, while a correlation of 0.0 would indicate no relationship. So 0.45 implies that total rushing yards is a moderately strong indication of how many games a team would win.

In comparison, total passing yards correlates with wins at 0.31. Compared with total rushing yards, total passing yards is less important in terms of winning games. This is what most fans and analysts notice when watching games or glancing at team stats. But does this mean that passing is less important than running in the NFL?

Before we settle on an answer, we need to consider the number of attempts of runs and passes. In baseball, the statisticians understood this 150 years ago when they created the batting average. If one player has 300 hits and another player has 350 hits, who is the better batter? The answer is we don't know until we divide the number of hits by their at-bats.

RUN AND PASS EFFICIENCY

How often have you heard an NFL commentator say, "When running back X gets at least Y carries, his team wins?" The clear implication is that the team in question should feed a steady diet of carries to the running back, and this will cause his team to win. What if we stated the same observation this way, "When his team is winning, running back X receives at least Y carries?" And by the way, why don't we ever hear, "When QB X passes at least Y times, his team wins?" Here's why:


Stat Win Correlation
Rush Attempts0.58
Pass Attempts-0.17


The negative correlation for pass attempts means that the more often a team passes, the less likely it is to win. The correlation of rush attempts with wins (0.58) is even stronger than that for total rushing yards (0.45). This is a curious result, and it’s where conventional NFL analysis begins to crumble.

When we see two things that appear correlated, it is natural for us to say that one causes the other. The runs come during the game, and the win comes at its conclusion. Therefore most fans and analysts assume the running causes the winning. The problem is, it usually doesn't. It's the winning that causes the running. Teams that are ahead, and likely to win, run the ball to take time off the clock and to minimize the risk of a turnover. Teams that are behind, and likely to lose, abandon the run in favor of the pass. Statistics can measure the correlation, but it can't determine the direction of causation.

The critical question then becomes: how can we truly measure a team's passing and running abilities and their respective contributions to winning? The answer is football's equivalent to the batting average--efficiency stats. If we want to know how good a team is at running, the best way is to know how many yards it tends to gain each time it runs the ball--yards per rush attempt. The table below lists basic passing statistics and their correlation with season wins.



StatWin Correlation
Pass Yards0.31
Pass Attempts-0.17
Pass Yds/Att0.61


Yards per pass attempt is merely pass yards divided by pass attempts. So we have a relatively weak statistic (0.31) divided by an even weaker one with a negative correlation with winning (-0.17). We would expect to have a fairly meaningless result, but we don't. Passing efficiency turns out to be strongly correlated with winning (0.61). And unless having a lead in a game ‘causes’ a team’s passes to be more successful, we can safely say that passing efficiency leads to winning.

Because sacks are an important factor in the passing game, I include plays that result in sacks as pass attempts for the purpose of calculating efficiency. Likewise, I also subtract sack yards from total passing yards. I call this true pass efficiency and have found it correlates better with both offensive points scored and wins.

Continue reading part 2 of the article.

How Good Will NE Be in ’07?

The Patriots are the consensus pre-season favorites to win the Super Bowl next February. The going odds that they will take home the Lombardi Trophy are, astonishingly, almost even at 9 to 5. The optimism in New England is due to the acquisition of three star players—wide receivers Randy Moss and Dante Stallworth, and linebacker Adalius Thomas.

MOSS AND STALLWORTH

The wide receiver position was seen as one of the Patriots' weaknesses last year, so the off-season upgrades are expected to dramatically improve their passing game. Unspectacular players such as Reche Caldwell and Jabar Gaffney became Tom Brady’s primary targets by the end of last season. The combination of a quarterback as good as Brady and receivers as talented as Moss and Stallworth could make New England’s passing attack dominant. But just how good can we expect them to become?

By replacing the performance of Patriots’ top two receivers of ’06 with the appropriate stats of Moss and Stallworth, we can estimate the most important measure of a team’s passing game—passing yards per attempt.

First, some assumptions: Tight ends and running backs will have the same performance as last year. Tom Brady will have his usual brilliant season. He’ll throw as few interceptions per attempt as he did last year, and he’ll be highly accurate with a 61.9% completion rate, which happens to be both his career average and last year’s rate. (Brady has been remarkably consistent. His best year was 2001 with a 63.9% completion rate, and his worst was 2003 with a 60.2% rate.) We’ll also assume both new receivers will remain healthy all year.

First, let’s look at Randy Moss. His stats over his last two seasons at Oakland are listed below. ADJ YDS is receiving yards adjusted for playing a full 16 game season.


 YR  G  REC  YDS   AVG   REC/G  ADJ YDS
2005 16 60 1005 16.8 3.8 1005.0
2006 13 42 553 13.2 3.2 682.3
Moss’s performance in 2006 was obviously hurt by playing with a sub-par team. Raiders quarterback Andrew Walter cannot be compared to Brady. Moss’s performance in 2005 was actually better than his final year in Minnesota, when he wasn’t even the top receiver for the Vikings. We’ll use his ’05 stats.

Despite all the attention on Moss, Dante Stallworth may prove to be the better acquisition. His stats over the last two years are below. He was in Philadelphia in ’06, and New Orleans in ’05.

 YR  G  REC  YDS  AVG  REC/G  ADJ YDS
2005 12 38 725 19.1 3.2 967.7
2006 16 70 945 13.5 4.4 945.0
Stallworth had impressive stats both years. After projecting his numbers for playing a full 16 game season in ’05, we’ll use the average of his stats for the last two years.

Last year, New England’s passing offense was mediocre. They ranked 11th in total yards, but more importantly, only 15th in passing efficiency with 6.81 yards per attempt. To see how much that stat can improve we’ll start by looking at the ’06 receiving corps. WRs with fewer than 3 catches are excluded to save space.

NAME             G REC  YDS  AVG  REC/G  ADJ YDS
Reche Caldwell 16 61 760 12.5 3.8 762.5
Doug Gabriel 12 25 344 13.8 2.1 460.0
Troy Brown 16 43 384 8.9 2.7 382.7
Jabar Gaffney 10 11 142 12.9 1.1 227.0
Chad Jackson 12 13 152 11.7 1.1 202.8
Now let’s replace the Patriot’s top two receivers with Moss and Stallworth. We’ll assume the #3 receiver remains Troy Brown, or at least will have stats very similar to his. We’ll also leave the 4th and 5th receivers as they are. Those receptions may go to Caldwell, Gaffney or Jackson, or to second tier acquisions Wes Welker and Kelley Washington, depending on who is healthy and active. But the resulting yardage should be similar for the #3 receivers and below on the depth chart. Below is the Patriot’s projected receiving corps for ’07.

NAME              G  REC  YDS  AVG  REC/G  ADJ YDS
Randy Moss 16 61 1021 16.8 3.8 1021.0
Dante Stallworth 16 61 991 16.3 3.8 991.0
Troy Brown 16 43 384 8.9 2.7 382.7
(Jabar Gaffney) 10 11 142 12.9 1.1 227.0
(Chad Jackson) 12 13 152 11.7 1.1 202.8
Where last year the Pat’s WRs contributed 1798 yards, next year’s squad should produce about 2706 yards, a 50% improvement. Add a steady 170 catches for 1792 yards from TEs and RBs for a total of 362 receptions. To achieve that number of receptions given a 62% completion rate would require 584 attempts.

The result is 362 catches for 4498 yards on 584 attempts, improving the Patriot’s pass efficiency from 6.81 to 7.70 yds/att. This would rank New England’s passing offense third in 2006 and 21st out of 160 in the last five seasons. But what does this mean in terms of wins?

By using a regression model to estimate regular season wins based on efficiency stats, we can compare how many wins the Patriot’s should expect given both last year’s and this year’s receivers.

The regression model uses each team’s offensive and defensive running and passing efficiency stats, plus turnover rates and penalty yards, to calculate a probability of the outcome of any given game. After calculating each game’s probability for the 2007 season, we can then sum the probability of each possible season outcome for the Patriots to estimate how many wins a team will have.

For the Patriots, we’ll run the process twice, first using the pass efficiency from ’06, and then with the projected ’07 pass efficiency. Here is how the ’07 season would look using the old stats:

Vis. Home P(V)  P(H)
NE NYJ 0.49 0.51
SD NE 0.57 0.43
BUF NE 0.24 0.76
NE CIN 0.47 0.53
CLE NE 0.12 0.88
NE DAL 0.41 0.59
NE MIA 0.50 0.50
WAS NE 0.19 0.81
NE IND 0.27 0.73
NE BUF 0.60 0.40
PHI NE 0.57 0.43
NE BAL 0.34 0.66
PIT NE 0.41 0.59
NYJ NE 0.32 0.68
MIA NE 0.32 0.68
NE NYG 0.56 0.44
Using the law of total probability, we can estimate the Patriots would win approximately 9 games this year, given their exceptionally difficult schedule. Here is the actual distribution:




But with their new receivers, New England can expect better probabilities. Here is their revised 2007 season and the associated total probabilities of season wins:

Vis. Home P(V)  P(H)
NE NYJ 0.58 0.42
SD NE 0.48 0.52
BUF NE 0.18 0.82
NE CIN 0.56 0.44
CLE NE 0.09 0.91
NE DAL 0.49 0.51
NE MIA 0.59 0.41
WAS NE 0.14 0.86
NE IND 0.34 0.66
NE BUF 0.68 0.32
PHI NE 0.49 0.51
NE BAL 0.42 0.58
PIT NE 0.33 0.67
NYJ NE 0.25 0.75
MIA NE 0.25 0.75
NE NYG 0.64 0.36


The Patriots are now projected to win 10 games instead of 9. The addition of Moss and Stallworth are worth 1 additional win. I expected a much larger difference, particularly because offensive pass efficiency is the factor with the most weight in the regression model. But another way to look at the improvement is to compare how the game-by-game probabilities changed. Notice that the addition of Moss and Stallworth are worth about a 9 point improvement in the probability to win each game. That’s an 18 point swing when considering the opponent’s win probability is equally reduced.

ADALIUS THOMAS

Projecting the improvement to the Patriot’s defense due to Adulius Thomas’s addition is tougher than for the WRs because it is much harder to quantify Thomas’s contribution to team defense. He is extremely versatile and athletic enough to impact every part of defensive football, from sacks to stopping the run to defending passes.

I propose a simpler but less exact approach. We will compute the share of Thomas’s contribution to his former team’s defense. The Ravens’ defense was historically good in 2006, ranking first or second in nearly every statistical category. We will improve the Patriot’s defensive stats towards those of the Ravens in proportion to Thomas’s share of overall team defense. In other words, if Thomas counted for half the tackles, assists, and sacks of the Ravens, then we would credit the Patriots with an improvement of 50% of the difference between Baltimore and New England’s team stats.

In 2006, The Ravens’ and Thomas’s stats were:

         TKL  AST  SCK  INT  PD  FF  FR
Ravens 707 200 60 28 111 12 11
Thomas 64 19 11 1 7 0 1
Thomas(%) 9% 10% 18% 4% 6% 0% 9%
Thomas recorded 9% of the Ravens’ tackles and 18% of their sacks. Excluding the forced fumble stat, the average share of Thomas’s contribution was 9%. To be generous, and to account for moving to a team with fewer stars where he can make a bigger impact, we’ll give Thomas credit for 10%.

The adjustment to New England’s defensive efficiency stats is calculated below. YPA is passing yards per attempt, YPR is yards per run, and INTPA is interceptions per pass attempt.

             YPA   YPR   INTPA
NE ’06 5.70 3.90 0.042
BAL ’06 5.29 3.30 0.055
Adjustment -0.04 -0.06 +0.002
NE ’07 5.66 3.84 0.044
Here is how NE’s season looks with their new WRs and Adalius Thomas.

Vis. Home P(V)  P(H)
NE NYJ 0.59 0.41
SD NE 0.47 0.53
BUF NE 0.17 0.83
NE CIN 0.57 0.43
CLE NE 0.09 0.91
NE DAL 0.50 0.50
NE MIA 0.60 0.40
WAS NE 0.14 0.86
NE IND 0.35 0.65
NE BUF 0.69 0.31
PHI NE 0.47 0.53
NE BAL 0.43 0.57
PIT NE 0.31 0.69
NYJ NE 0.24 0.76
MIA NE 0.24 0.76
NE NYG 0.65 0.35

Thomas’s contribution is significantly less dramatic than that of Moss and Stallworth. He counts for about 1 additional point of probability per game, for a 2 point swing, and adds far less than one win to the Patriot’s projections.