Week 15 QB Ratings Adjusted for Pass Defense

In this post I'll recap how my QB rating is determined, and I'll add a new dimension--adjustments for opposing pass defenses.

QB Rating Recap

The QB rating I post often is different from most, including the NFL passer rating. The stat I use to compare QB performance is +WP16 or wins added per 16 games. +WP16 estimates the number of wins above average a quarterback's performance would mean to his team over a full regular season.

The +WP16 stat estimates the wins added based on a regression that analyzes what makes teams win. The QB stats included are passing yards per attempt, rushing yards per attempt, sack yards per attempt, fumble rates, and interception rates. +WP16 isolates QB performance by assuming an average rushing game and an average defense.

First downs and touchdowns are not included despite their apparent importance because they are intermediate outcomes between actual QB performance and the outcome of interest--team wins.

Perhaps the most unique aspect of +WP16 is that it does not include passing yards gained by receivers after the catch (YAC). Instead it is based on air yards--the yards the ball travels through the air forward of the line of scrimmage. The percent of passing yards made up of YAC varies widely from QB to QB. Brodie Croyle gets 68% of "his" yards from YAC while Ben Roethlisberger gets only 34% of his yards from YAC.

There is some debate over whether or how much YAC should be credited to QBs. In my mind, the answer is much closer to none than to all.

There is statistical evidence that YAC belongs to the receiver. Year-to-year correlations of YAC are far stronger for receivers than for QBs. Accuracy rates and QBs' abilities to read the defense and "see the field" (measured by interception rates) do not contribute to YAC. It appears that YAC is determined by receiver ability far more than QB ability.

The types of passes thrown by a QB also help determine YAC. Screen passes and check downs to RBs, which are not difficult passes, tend to produce more YAC relative to air yards. Deep out passes, which are more difficult and require more QB skill, tend to generate more air yards and often no YAC at all. The other type of pass that can create a lot of YAC is when a WR beats the defense deep by simply out-running them. Although it certainly helps if a QB can get the ball to a receiver who has gotten behind his coverage, those types of plays are rare and are really made possible by the WR, at least in my mind.

For the reasons above, I believe the truer measure of a QB's performance is one that does not include YAC. Here is the ranking of QBs based on performance so far this season:

Rank	Name	QBRat	Att	Yds	Int	Rush	Yds	Sk Yds	Fum	AY/A	%YAC	+WP16
1	Brady	119.7	503	4235	6	29	91	107	4	4.9	42	2.48
2	Garrard	101.6	307	2310	2	45	166	91	3	4.8	37	2.21
3	Manning P	95.2	464	3634	14	19	-4	122	5	4.9	37	1.29
4	Anderson	85.8	459	3384	14	30	64	100	5	4.5	38	0.95
5	Romo	101.0	462	3868	17	25	116	166	9	4.9	41	0.89
6	Schaub	87.2	289	2241	9	17	52	126	7	5.0	36	0.87
7	Roethlisberger	104.1	404	3154	11	35	204	347	9	5.2	34	0.86
8	Cutler	90.8	398	3096	12	41	163	119	8	4.3	44	0.46
9	Garcia	93.6	307	2244	4	34	116	98	3	3.7	50	0.44
10	Palmer	86.2	522	3700	17	22	11	119	5	4.3	40	0.36
11	Hasselbeck	92.0	510	3620	10	35	68	183	8	4.0	44	0.17
12	Brees	92.1	550	3819	15	22	53	89	7	3.8	45	0.11
13	Favre	97.7	492	3905	13	25	-9	89	8	3.8	52	-0.16
14	Warner	87.6	360	2748	15	13	-4	140	11	4.7	38	-0.22
15	Kitna	84.6	497	3707	17	21	55	304	15	4.7	37	-0.26
16	McNabb	86.8	397	2716	6	43	199	192	7	3.4	50	-0.33
17	Campbell	77.6	417	2700	11	36	185	110	13	3.6	44	-0.44
18	Jackson	69.6	222	1516	10	42	180	47	3	3.6	47	-0.58
19	Pennington	85.8	228	1501	7	18	27	142	3	4.1	38	-0.61
20	Boller	75.2	275	1743	10	19	89	159	5	4.0	37	-0.74
21	Young	70.1	342	2223	16	82	375	132	7	3.5	46	-0.92
22	Edwards	75.6	213	1336	5	9	25	71	2	3.1	51	-0.94
23	Bulger	71.4	353	2216	13	9	13	248	6	4.1	35	-1.11
24	Manning E	72.6	482	2974	17	24	56	186	7	3.5	43	-1.11
25	Losman	76.9	175	1204	6	20	110	103	4	3.4	50	-1.21
26	Rivers	80.0	412	2828	15	26	31	153	10	3.7	46	-1.22
27	Culpepper	78.0	186	1331	5	20	40	130	8	4.0	44	-1.24
28	Harrington	77.2	348	2215	8	14	33	192	0	3.0	52	-1.26
29	Smith	57.2	193	914	4	13	89	121	6	3.1	35	-1.43
30	Lemon	69.9	247	1479	6	25	86	120	6	3.0	50	-1.52
31	Clemens	59.0	225	1414	10	19	93	125	4	3.4	46	-1.80
32	McNair	73.9	205	1113	4	10	32	85	8	2.8	49	-2.02
33	Huard	72.6	296	1952	13	9	-1	204	4	3.6	45	-2.03
34	Grossman	66.4	225	1411	7	14	27	198	6	3.5	44	-2.03
35	Griese	75.6	262	1803	12	13	28	114	6	3.3	52	-2.22
36	Testaverde	65.8	172	952	6	8	23	46	3	2.6	52	-2.24
37	Frerotte	62.0	162	1010	11	6	3	59	3	3.9	38	-2.34
38	McCown	68.6	182	1105	11	29	143	78	10	3.0	51	-3.14
39	Croyle	71.7	169	963	5	5	16	72	4	1.8	68	-3.57
40	Dilfer	55.1	219	1166	12	10	25	182	8	3.0	43	-3.87
Rank	Name	QBRat	Att	Yds	Int	Rush	Yds	Sk Yds	Fum	AY/A	%YAC	+WP16

The biggest surprise at the top of the list is David Garrard. He's ranked so high primarily because of his phenomenally low turnover rate, having thrown only three interceptions all year. His air yards per attempt (AY/A) is impressive too. He ranks among the elite QBs in the league with 4.8 yds/att. In total, Garrard ranks second behind Brady with +2.21 wins added.

Adjusting for Opponents' Pass Defense

When comparing team performance, opponent strength matters a great deal. Schedules in the NFL are not balanced. During the regular season a team only plays 13 of the 31 other teams in the league, 3 of them twice. If those 3 teams a team's own division are among the toughest, and the 18 teams it never plays comprise some of the weakest opponents, then the team would appear to be much worse than it would be if it had an average schedule.

Early in the season opponent adjustments are most important, because as the season goes on and each team faces more different types of opponents, each team's strength of schedule tends to even out. But not completely.

It stands to reason that QB performance ratings would also be affected by opponent strength, specifically opponent pass defense. To confirm this effect, each team's strength of pass defense was computed. That includes average pass yards per attempt allowed, sack yards per pass play, and interception rate. Each stat was weighted according to how important they are towards winning based on the same regression model that determine the weights of the QB rating. This produced a unit-less pass defense score.

Then each QB's average opponent pass defense strength was determined. QBs were counted as having played a team if they had more than 10 attempts in a game.

To see how opposing pass defenses affect QB performance, the correlation coefficient between each QB's rating and his opponents' average pass defense strength was calculated. The correlation was 0.098, which is weak but what we'd expect. About 9% of a QBs season-long performance can be attributed to his opponent's pass defense. Or strictly speaking, one standard deviation (SD) increase in opposition strength results in one SD reduction in QB performance.

The last step in the process was to adjust the QB rating (+WP16) accordingly. Each QB's rating was adjusted by 0.098 SDs for each SD of opponent strength. The stronger the average opponents' pass defense, the higher the adjustment for the QB.

The table below lists each QB's unadjusted rating, his average opponent's pass defense strength, and the adjusted rating. To make the numbers a little easier to understand, pass defense strength is converted into GWP (generic win probability), or the probability a pass defense would win a game against a league-average opponent given an average run defense and offense.

Rank	Name	+WP16	Opp Pass D Str	Adj +WP16
1	Garrard	2.21	0.56	2.53
2	Brady	2.48	0.50	2.48
3	Manning P	1.29	0.53	1.43
4	Schaub	0.87	0.54	1.06
5	Anderson	0.95	0.49	0.87
6	Romo	0.90	0.49	0.86
7	Roethlisberger	0.86	0.47	0.69
8	Cutler	0.46	0.53	0.60
9	Garcia	0.44	0.53	0.58
10	Palmer	0.36	0.51	0.40
11	Brees	0.11	0.54	0.34
12	Hasselbeck	0.18	0.47	0.04
13	Favre	-0.16	0.50	-0.14
14	Kitna	-0.26	0.52	-0.15
15	Warner	-0.22	0.49	-0.27
16	McNabb	-0.33	0.51	-0.27
17	Campbell	-0.44	0.50	-0.43
18	Jackson	-0.58	0.49	-0.64
19	Pennington	-0.60	0.49	-0.65
20	Boller	-0.74	0.51	-0.66
21	Young	-0.91	0.52	-0.80
22	Edwards	-0.94	0.48	-1.02
23	Culpepper	-1.24	0.54	-1.04
24	Bulger	-1.10	0.51	-1.07
25	Harrington	-1.26	0.53	-1.08
26	Rivers	-1.22	0.52	-1.10
27	Manning E	-1.11	0.50	-1.12
28	Losman	-1.21	0.47	-1.36
29	Smith	-1.43	0.51	-1.38
30	Lemon	-1.51	0.48	-1.62
31	Clemens	-1.80	0.50	-1.80
32	Grossman	-2.03	0.54	-1.82
33	Huard	-2.03	0.52	-1.91
34	Testaverde	-2.24	0.54	-2.04
35	Griese	-2.22	0.53	-2.08
36	McNair	-2.01	0.44	-2.34
37	Frerotte	-2.33	0.50	-2.34
38	McCown	-3.14	0.50	-3.16
39	Croyle	-3.57	0.51	-3.50
40	Dilfer	-3.87	0.51	-3.82

David Garrard

At this point in the season (going into week 16), we wouldn't expect there to be much change in the ratings by accounting for opponent strength. Most teams have faced some good pass defenses and some weak ones. And indeed there is relatively little change in the rankings.

There are some notable exceptions, however. The QB who faced the weakest pass defenses was Steve McNair who played against CIN, ARI, CLE, and SF, all of which had below average pass defenses. McNair still managed to turn in a miserable performance in 2007 due to age, injury, or probably both.

The other notable exception is David Garrard. Garrard faced the toughest pass defenses of any QB in 2007. If not for an injury that kept him out of three games against TB, NO, and TEN in weeks 8,9, and 10, he would have an even tougher year. Despite his uphill battle, Garrard is playing extremely well this season. Before adjusting for opponent strength he ranked second among all QBs (not even counting for his week 17 demolition of OAK.)

In the year of Tom Brady and the Patriots, it may be David Garrard who is the best QB this season. After factoring in opponent strength, his +WP16 rating jumps from +2.2 to +2.5, edging Brady for the top spot. That's remarkable considering Garrard wasn't even his team's starter in training camp. Jaguars coach, Jack Del Rio, looks like a genius for cutting Leftwhich and sticking with Garrard.