Yesterday (the Saturday before Super Bowl XLII) the Boston Herald reported that a source alleged "a member of the [Patriots] video department filmed the Rams’ final walkthrough" before the 2002 Super Bowl in which the Patriots upset the heavily favored Rams. ESPN reported that the Rams stated the walkthrough primarily focused on plays they intended to run in the red zone.
Asked about the allegations yesterday, commissioner Robert Goodell answered, "I’m not aware of that.” NFL Spokesman Greg Aiello added, "We have no information on that."
Then later, on the same or very next day, spokesman Aiello told the AP, "We were aware of the rumor months ago and looked into it. There was no evidence of it on the tapes or in the notes produced by the Patriots, and the Patriots told us it was not true." (emphasis mine)
Well, that clears that up. Imagine if the movie Untouchables ended this way: Elliot Ness--"Your honor, there is no evidence of tax evasion in the documents provided by Mr. Capone, and he has told us the charges are not true." Judge--"Case dismissed."
Previous Research
I don't normally add my own opinions here at this site, but I'll make an exception here because of my previous look into possible statistical evidence that the Patriots benefited from unfair advantages. Specifically, the Patriots had won about 2 more games per year, every year, from 2002-2006 than their on-field performance would statistically indicate. In other words, other teams with similar performance stats win 2 fewer games in a season than Belichick's Patriots did. If the Patriots used knowledge of their opponents' play calls in primarily high-leverage situations (3rd downs or critical 4th quarter plays) we would see this kind of result.
My own interest in statistics began when I did my masters thesis, a research paper on midshipmen at the U.S. Naval Academy who violated its Honor Concept. It was basically research on cheaters at an elite institution in highly competitive and stressful environment. Although not the primary focus of my research, along the way I learned that cheaters are recidivists. Once they are able to rationalize their behavior, they will continue to cheat. Additionally, those who are caught are rarely nabbed on their first attempt, most likely because they select methods and opportunities hard to detect. They go out of their way to hide their cheating activity. No surprise there.
(Coincidentally, it was at Annapolis where Belichick learned his football under his dad, an assistant coach for Navy.)
So when we saw first hand how the Patriots violated league rules, I surmised it was highly unlikely that their activities were limited to taping defensive signals in isolated games. I think it would be naive to believe otherwise. Several very smart commenters (with very good points) accused me of making "assumptions" about the Patriots' cheating. So if the recent allegations have any merit, I'd feel somewhat vindicated.
Super Bowl XXXVI
Then, just as I was thinking of writing this post this afternoon, I channel-surfed onto the NFL Films highlights of the Patriots-Rams 2002 Super Bowl on ESPN2. Immediately prior to the drive in which Ty Law jumped a quick out route to intercept a Kurt Warner pass and return it for a touchdown, there was a sideline shot of three Patriot defenders discussing signals. (My thanks to TiVo, by the way.)
In the shot, Safety Lawyer Malloy runs up to cornerbacks Terrel Buckley and Terrance Shaw and says,
"Listen! Listen!
We got 'Sloop.' (makes a hand signal)
We got 'Move'...you know the move signal. (makes a different signal)
We got 'Marine 5'...'Marine.' (makes signal)
We got 'Seagull.' (another signal)
We got, this is...this is 'Double Out' right here. (making signal)"
Buckley and Shaw mimic the signals and nod each time.
My guess is these would not be their own signals--they would know them already and Milloy's words "we got" and "you know" plus the names for each signal suggest the signals are somewhat but not entirely new. Additionally, "double out" sounds like an offensive call. They appear to be rehearsing the Rams' signals, although possibly Warner's QB signals and not sideline signals. But the main point is that knowledge of some of the Rams' offensive signals was widespread on the Patriots defense, it was a priority to them, and it apparently didn't hurt New England's performance.
On the other hand, I'd guess that all teams try to read QB signals, but if they could do it reliably well, offenses wouldn't use them. Offenses would also be able to use countermeasures or easily spoof a defense. The Patriots may just play this part of the game better than other teams. If so, we'd see the same statistical results I found in my earlier post. It also underscores that outsiders like myself really don't have a any idea of what really goes on inside the film rooms and coordinator booths in the NFL.
Belichick's Focus
Perhaps Belichick had an intense focus, within the rules, on exploiting opponent's signals and deceiving them with his own. In the military we call this 'SigInt' for signals intelligence, a critically important part of modern warfare. The advantage from signal exploitation may have encouraged the Patriots to pursue it beyond permitted means. Jets coach and former Belichick assistant Eric Mangini would have been aware of the importance of this part of the game to the Patriots, so It's no surprise he was the one to blow the whistle.
A Real Investigation
The other point is that if the NFL really wanted to investigate these things, there is ample evidence in the NFL Films archive. There are probably hours upon hours of sideline film from just the Patriots' Super Bowls alone, not to mention playoff games or regular season games. An honest investigation would have taken weeks, not the couple of days the NFL took before destroying the evidence.
Don't get me wrong. I'm not a Belichick hater. I appreciate his cerebral approach to the game. I like how he goes for it on 4th down and focuses intensely on details, and I don't find him arrogant at all. But I do have a strong intolerance for cheating, and I believe these things deserve to be investigated.
More Patriots Cheating Allegations
The Passing Premium
In previous posts, I've referred to a concept called the passing premium. Specifically, I point to Benjamin Alamar's paper which found that the expected yards per play is higher for passing than for running. The difference accounts for incompletions and the risk of interception.
I've discovered a flaw in the author's analysis, however. He does not appear to account for sacks.
His analysis finds that for every passing play in the 2005 NFL season, the expected gain is 5.8 yards per attempt. Interceptions are factored in by assigning them a -45 yard value. (40-50 yards as an equivalent for interceptions is a commonly accepted value. It also makes intuitive sense because an interception differs from an incompletion by precluding the possibility of a punt, which usually nets about 40 yards.) Touchdown passes also get an adjustment because the goal line truncates the pass. For every touchdown pass, an extra 10 yds is added.
The true expected gain from a pass play should be:
(Pass Yds Gained - Sack Yds - Int Adjustment + TD Adjustment) / (Pass Att + Sacks)
The author leaves out the sacks both in the numerator and denominator, which makes a difference.
The average run yields 4.1 yards. The difference is an unexplained premium for passing, suggesting that play selection is not rationally balanced in the NFL.
Realizing that football is more complex than a binary run or pass decision, and that averages are not always the truest measure of performance in all situations, the difference of 1.7 yards per play remains considerable. So despite those limitations, perhaps coaches should be calling more passes and fewer runs.
I performed my own analysis, repeating Alamar's methodology for 2005 data, and then expanding it to data from the 2002-2006 seasons. By adding 10 yds per touchdown pass and subtracting 45 yards per interception, I also calculated 5.8 yds per attempt. But when I subtracted sack yards, the expected yield for a pass attempt becomes 5.0 yards per attempt.
The passing premium now becomes 5.0 - 4.1 = 0.9 yards per play, a smaller difference than the author found.
I also calculated running yards per attempt when the same touchdown adjustment is applied. Aren't many running touchdowns truncated by the end zone too? It does seem generous to add 10 yards because some touchdown runs are goal line dives, but many are not. Some touchdown passes would not automatically yield an extra 10 yards either. Adding the 10 yard touchdown bonus makes the expected gain for running 4.5 yards per attempt.
The passing premium would now become only 5.0 - 4.5 = 0.5 yards per play.
Running in some situations, however, has value in addition to yards gained. Towards the end of a game, the leading team can use more clock time by running, denying additional opportunities for the trailing team to score. In short yardage situations, running for a short gain can be more beneficial than the chance to have a longer gain with a pass. Goal line runs can sometimes require 2 or 3 attempts before a gain of the single yard that yields the touchdown, but that single yard is worth the possibility of no gain on previous plays. (In a way, the generous 10 yard bonus for a touchdown run seems more appropriate considering the frequent stuffs on the goal line due to the high predictability of running in that situation.
All things considered, perhaps the run and pass are nearly balanced in the NFL. Balance is important because it suggests maximization. If a team runs too much, a defense can concentrate their efforts on stopping run plays, reducing the expected gain for a run but a greater expected gain for pass plays. Every team would have their own optimum balance, but over the league as a whole the optimum run-pass mix would yield about the same expected gain every play.
Underdogs, Reducing Possessions, and Super Bowl XLII
In the last article, I made the point that underdogs have a better chance of winning a game when each opponent has fewer possessions. Specifically I wrote, "The more possessions each team has, the more likely it is the better team is going to eventually come out on top. With fewer total possessions, the underdog has a better chance to win because randomness plays a bigger role relative to team ability in a game’s outcome." This article will estimate just how much an underdog can benefit by reducing the total number of possessions in a game.
I built a crude simulation of a football game in the PHP programming language. By specifying the number of possessions for both teams and the scoring rate of each team, a simulated score and winner can be determined. By running the simulation many times, we can get a good estimate of the win probability for various numbers of total possessions.
But first, we need to pick a couple of teams as guinea pigs. One needs to be a big underdog against the other. Hmmm, let's see. How about the Giants and Patriots?
The Patriots scored touchdowns in 42% of their possessions in 2007. And 13% of their possessions resulted in field goals. They gave up TDs in 17% of their opponents' possessions and allowed FGs in 7% . In contrast, the Giants scored TDs in 21% of their possessions and kicked a FG in 12%. They allowed TDs in 19% of their opponent possessions and gave up FGs in 11%. The table below summarizes each team's respective drive stats.
| Scoring Rate |  NE (%) |  NYG (%) |
| Own TDs | 42 | 21 |
| Own FGs | 13 | 12 |
| Opp TDs | 17 | 19 |
| Opp FGs | 7 | 11 |
If we assume that the each team will score according to the mid-point between each offense's and defense's scoring rate, we can construct a fairly solid model. For example, given the Patriots' 42% offensive TD rate, and the Giants' 19% opponent TD rate, we could estimate the Patriots would score (42 + 19) / 2 = 30.5% of the time. (I realize this is pretty rough, but it suits the simulation's purpose.)
Unfortunately for New York Giants fans, the Patriots won the first simulation 38-7. That was for 12 possessions for each team, the most common number of team possessions in the NFL. But one simple simulation is pretty pointless. After 10,000 of them however, New England won 75.6% of the games and the Giants won 20.5%, with 3.8% of the games going into overtime.
But what if each team only had 10 possessions? How do the underdogs fare? The Patriots' win 72.7% of the games and the Giants win 22.4%. Reducing the number of possessions does boost the chances of the underdog, but only slightly.
The table below lists typical numbers of possessions for each team in NFL games along with the simulated probabilities of winning for each team.
| Possessions | NE Wins | NYG Wins | Overtime |
| 9 | 71.4 | 23.5 | 5.1 |
| 10 | 72.7 | 22.4 | 4.9 |
| 11 | 74.0 | 21.5 | 4.5 |
| 12 | 75.5 | 20.7 | 3.8 |
| 13 | 76.5 | 20.0 | 3.5 |
First thing to note is that the Giants have an improbable challenge, no matter how few possessions to which they can limit their opponents. This method appears to confirm my standard logistic regression efficiency model that gives the Giants about a 1 in 4 shot at the title. But that's nothing to sneeze at. How much sleep would you get if you knew you had a 25% chance of your life savings being wiped out by morning? It's not a guaranteed victory for New England by any stretch.
The fewer the number of possessions, the greater the chance of upset. Letting the clock run is in the Giants' interest. Did you hear that Plaxico? Take the hit and stay in bounds, (as long as the score is close). You'll be helping your team more than you could ever understand.
Keeping an Offense "Off the Field"
I frequently hear this nugget of wisdom from football analysts: “Team X needs to run the ball to keep Team Y’s high-scoring offense off the field.” At first it makes sense. An offense can’t score if it’s not on the field, no matter how good it is. But at second glance, maybe it’s not so logical.
Number of Possessions
Unlike a sport such as hockey, football is mainly a game based on turns. Football has no face-offs or tip-offs. After one team has possession of the ball, regardless of how that possession ends, the other team will have a turn with possession of the ball. In a pure turn-based game, both teams are going to have an equal number of possessions. But football is complicated by the fact that time expires at the end of each half. In each half it is possible for the team that started the half with the ball to have one more possession than the other. Therefore, in almost all cases the game will end with the number of possessions being equal or within one possession.
I say almost all cases because there are rare exceptions. An interception or punt returned for a touchdown technically does not count as a possession. In those cases, it is possible for a team to have two consecutive possessions. But you don't have to be a statistician to know that's not a good way to get additional possessions. The analysis below accounts for such cases.
I’m not saying that clock management is not important. Once the game clock winds down towards 3 or 4 minutes remaining, it may become somewhat clear how many possessions are likely left for each team. At this point it makes sense to try to manage the clock by play selection and time-outs. And toward the end of a close game it’s absolutely critical.
But until the final minutes of a half, the way the possessions will shake out is completely unpredictable. Until that point, it’s simply silly to try to keep an offense “off the field” by running the ball. Chewing up time by running early in a half does not guarantee your team will be the one with the extra possession. Ironically, a coach may be keeping his own offense off the field for all he knows.
Slowing Down the Game
On the other hand, a strategy that slows down the game could benefit an underdog by causing fewer total possessions by both teams. The more possessions each team has, the more likely it is the better team is going to eventually come out on top. With fewer total possessions, the underdog has a better chance to win because randomness plays a bigger role relative to team ability in a game’s outcome. A 17-10 lead is far more vulnerable than a 34-20 lead to a single kick return or interception return for a TD.
Can Play Selection Really Reduce Possessions?
Can play selection really limit the number of opponent possessions, and if so by how much? Using data from the 2005-2006 seasons, I analyzed how team possessions and time of possession were affected by play selection.
Play selection can be defined multiple ways: run/pass ratio, runs as a percent of total plays, or a simple difference between runs and passes. I chose the simple difference (runs minus passes) because choosing a run excludes a pass. Every choice precludes the other option. A team can’t increase the number of running plays without declining to pass. (It also had the strongest correlation with time of possession (TOP).)
I adjusted the number of offensive possessions by the number of defensive touchdowns allowed. This correction is necessary due to the effect noted above in which a team can actually have two consecutive possessions due to a return for a touchdown. Likewise, opponent possessions were corrected for defensive touchdowns scored.
The correlations between play selection and the other variables are listed below. There are small but significant effects on TOP and team possessions due to play selection.
| Run/Pass Selection Correlation with: | Coefficient |
| Time of Possession | 0.23 |
| Adj Own Possessions | -0.16 |
| Adj Opponent Possessions | -0.15 |
The correlation is just as strong for a team’s own number of possessions as for opponent possessions. No surprise here—as we expected, the total number of possessions are reduced but the team choosing to run does not have an advantage.
But how strong is the effect? Is it worthwhile to sacrifice optimum play selection to reduce the number of possessions? Based on the standard deviations and correlations of each variable, we can estimate that for every 3 plays called as a run instead of a pass, a team can expect an average of 1:28 additional TOP.
The correlation between TOP and opponent possessions is 0.08. Therefore on average, a team with an extra 1:28 in TOP can reduce an opponent’s number of possessions by 0.64 drives. Therefore, to reduce an opponent’s number of possessions by a full possession, a team would need to swap about 5 passes for runs.
Teams average 28 run attempts and 35 pass attempts per game. That means a team would need to run about 33 times and pass about 30. Actually, it would probably be closer to 31 runs and 28 passes because of the reduction in remaining game time available.
A team would either need to have a very large advantage in the running game or already have a comfortably large lead to run so much. For even the worst passing teams, passes normally have a much higher expected return than runs. Accounting for the possibility of interceptions, passing still has a significantly bigger expected payoff. Except in specific circumstances such as short yardage situations, for every run play chosen over a pass play, a team is sacrificing some amount of total effectiveness (as long as a team runs often enough to prevent the defense from only defending the pass).
I think a much better strategy for reducing the number of opponent (and total) possessions would be to just tell your receivers and backs to take the hit and stay in bounds a few times. Reducing penalties would have a similar effect by keeping the clock running when it would otherwise stop. That way an underdog can keep the game relatively close by reducing total possessions and still have the freedom to optimize its play calling based on match-ups and game situations.
A coach should play calls to maximize his chances of keeping the ball, getting first downs, and scoring. After all, it’s only incomplete passes that stop the clock. If a coach is calling plays expecting a disproportionate number of incomplete passes, he’s probably going to lose anyway.
Is "Red Zone" Performance Real?
Team performance in the red zone is obviously very critical. Teams can make up for a lot of deficiencies if they can move the ball inside the opposition's 20 yard line. But is the red zone real? Certainly, it's tougher to move the ball efficiently inside the 20 because the field is compressed and the defense has less territory to protect. As a consequence, outside the red zone the 2007 league average pass completion rate was 63%, but inside the red zone it was only 56%. So in that sense, the red zone is real.
In this article, I'll compare 2007 quarterback performances inside and outside the red zone. After testing the differences statistically, we'll see if some QBs really have a knack for success inside the 20, or if we're just witnessing the randomness of a small subsample of passes.
Football experts saw a correlation in red zone performance and scoring, and quickly focused a lot of attention on how well a team does inside the 20. It does make sense, to a degree. Teams that are otherwise efficient and can move the ball, but don't put it in the end zone won't score and won't win. A quarterback whose stats are particularly good in the red zone is considered a clutch performer and is believed to have some special ability to perform when it counts most. Likewise, a QB who under-performs in the red zone is seen as lacking what it takes to succeed in the NFL.
"My theory is that the very same abilities that lead to success in the other 80 yards of the football field also lead to success inside the 20."
But what if red zone performance was really just a random subset of overall performance? If we arbitrarily divided the field into any other 20-yard segment and analyzed performance, would we find that some QBs have a special talent between the 40s? After all, pass attempts inside the red zone comprise only 13% of all passes.
My theory is that the very same abilities that lead to success in the other 80 yards of the football field also lead to success inside the 20. A QB who is accurate, aware, can read defenses, and has a strong arm will likely do well in any part of the field. Someone would have to convince me that there is some special talent that becomes more important in the red zone.
Some might say that abilities or flaws are magnified in the red zone because of the compressed field--the density of pass defenders is much higher. If that's true, we should see quarterbacks with good stats do especially well, and below-average quarterbacks should do especially poorly in the red zone.
But that's not what we find. The #1 QB in completion percentage in the red zone last year was Sage Rosenfels. Outside the red zone however, he ranked 20th. Trent Edwards was 29th outside the red zone, but 3rd inside. Looking at the rankings, they seemed very random.
Although completion percentage is generally not the best measure of QB performance, it might be particularly relevant in the red zone for several reasons. Yardage measures are problematic because passes thrown into the end zone are truncated at the goal line. Efficiency measures may not make sense because a 2 yd pass completion from the 2 on 3rd and goal is more meaningful than a 4 yd completion from the 18 on 3rd and 8. Over the course of an entire season and across a full 100-yard field, yards per attempt might tell us a lot more than within a select set of passes near the goal line. In general, a completion in the red zone is always better than an incomplete pass or an interception. Close to the goal line, there isn't often a better or deeper option--any completion is good.
To test whether red zone completion percentage is a special talent or simply a random subsample of overall completion percentage, I used a statistical test known as a t-test. Those familiar with statistics and regression know this is the test that indicates if a variable is significant or not. But it can be used on its own without a regression to test if observed differences are due to a systematic effect or just due to random variation or sample error.
The t-test produces a probability that the observed differences are really just due to chance. Generally, a p-value below 0.05 means a variable is significant--there is a 95% chance there really is a systematic connection between variables. For example, if Matt Cassel completed 4 out of 5 passes in one series of relief for Tom Brady, does that make him more accurate than Brady's 398 for 578? The t-test says no--the sample size is too low and the difference is not big enough to conclude Cassel is more accurate than Brady.
So I performed a t-test on each quarterback's completion percentage inside and outside the red zone. But there was a wrinkle. We already know it's tougher on every quarterback in the red zone because of field compression. We're really interested in whether some QBs significantly over- or under-perform their overall completion percentage. To compare apples to apples, I used a special version of the test that accounts for known expected differences in the means of each group. The NFL average completion percentage is 7.2 points lower for the red zone than outside the red zone, so I used 7.2% as the difference in means.
The table below lists the leading QBs in completion percentage. Their completion percentage in the non-red zone (NRZ) and inside the red zone (RZ) is listed. Next is listed a "value over average" (VOA) number that indicates how a QB over- or under-performed his overall percentage when inside the red zone, accounting for the general increase in difficulty inside the 20. A high positive number means a QB did especially well in the red zone compared to outside it. The final column is the result of the t-test. A number below 0.05 is generally considered significant.
Click on the table headers to sort.
| QB | Team | NRZ Pct | RZ Pct | RZ VOA | T-test |
| Anderson |  | 58.1 | 45.5 | -10.6 | 0.43 |
| Boller |  | 62.0 | 55.3 | -0.8 | 0.75 |
| Brady | | 70.6 | 61.6 | 5.5 | 0.86 |
| Brees |  | 66.9 | 71.4 | 15.3 | 0.04 |
| Bulger |  | 57.8 | 64.7 | 8.6 | 0.12 |
| Campbell |  | 61.0 | 52.1 | -4.0 | 0.84 |
| Clemens |  | 53.9 | 39.4 | -16.7 | 0.62 |
| Cutler |  | 64.4 | 56.9 | 0.8 | 0.98 |
| Edwards |  | 54.8 | 67.9 | 11.8 | 0.04 |
| Favre |  | 67.6 | 59.4 | 3.3 | 0.93 |
| Garcia |  | 65.0 | 53.3 | -2.8 | 0.63 |
| Garrard |  | 65.1 | 58.9 | 2.8 | 0.81 |
| Griese |  | 63.4 | 46.7 | -9.4 | 0.34 |
| Harrington |  | 62.8 | 50.0 | -6.1 | 0.56 |
| Hasselbeck |  | 63.7 | 55.4 | -0.7 | 0.90 |
| Huard |  | 62.6 | 56.7 | 0.6 | 0.89 |
| Jackson |  | 59.5 | 44.0 | -12.1 | 0.42 |
| Kitna |  | 64.1 | 56.1 | 0.0 | 0.91 |
| Manning |  | 67.3 | 56.0 | -0.1 | 0.55 |
| Manning | | 56.0 | 56.8 | 0.7 | 0.19 |
| McNabb |  | 63.9 | 47.8 | -8.3 | 0.19 |
| Palmer |  | 65.8 | 59.3 | 3.2 | 0.85 |
| Pennington | | 70.7 | 54.8 | -1.3 | 0.35 |
| Rivers |  | 62.4 | 47.8 | -8.3 | 0.28 |
| Roethlisberger |  | 66.9 | 58.3 | 2.2 | 0.91 |
| Romo |  | 66.1 | 53.6 | -2.5 | 0.86 |
| Rosenfels |  | 62.6 | 76.9 | 20.8 | 0.04 |
| Schaub | | 68.0 | 53.3 | -2.8 | 0.43 |
| Warner |  | 62.9 | 58.3 | 2.2 | 0.67 |
| Young |  | 63.6 | 50.0 | -6.1 | 0.46 |
Note that there are in fact three QBs with statistically significant differences in completion percentage inside the red zone. Brees, Edwards, and Rosenfels all had significantly better than expected performance in the red zone. No QBs were significantly worse than expected. So this is proof that those three QBs have a special ability to perform in the compressed field inside the 20, right?
"It appears that despite all analysis to the contrary, there is nothing special about any particular quarterback's ability inside the red zone as compared to outside the 20."
From this, it appears that despite all analysis to the contrary, there is nothing special about any particular quarterback's ability inside the red zone as compared to outside the 20. It's simply a random subsample of overall performance. If a QB is a good passer, he'll probably be good inside the 20.
If this conclusion is correct, it should adjust our view of some of the QBs in the table above. (Sort by the red zone value over average (RZ VOA) column to see the most over- and under-performing QBs.) Take Trent Edwards for example. His over-performance inside the red zone probably makes him appear to be more productive a passer than he truly is. He's not likely to over-perform inside the 20 in 2008 like he did in 2007. Chances are Buffalo fans will be disappointed next year.
The opposite could be said of QBs such as Kellen Clemens. He under-performed in the red zone in 2007. Although his overall numbers aren't terribly great, chances are he will not under-perform inside the 20 as severely as he did last season. It's not a guarantee Clemens will improve, but he would likely trend toward his overall performance level and not repeat his dismal 39% completion rate in the red zone.
This result also means that prediction models and handicappers that overweight red zone performance are reducing their predictive power. They are chasing the past randomness of a small subset of performance. The same is probably also true of the myriad of stats "splits" such as "under the lights" or "road division games " or any other silly and arbitrary classification.
Cold Weather Effect on Scoring
In a recent post I theorized that the sudden importance of run defense in the playoffs might be due to cold weather. This post will continue that line of analysis and look at the effect of cold weather on scoring.
The past weekend's conference championship games were played in frigid weather. It seemed that expert after expert remarked that cold weather would keep the scoring down. Certainly it makes sense to anyone who's played sports in extremely cold weather. It definitely makes it harder to throw, catch, and even kick. But it's just as cold for defenses as for offenses. So does cold weather really keep NFL scores lower?
Here are the average home and visitor scores for various circumstances. The first column is for all regular season games in the 2002-2006 seasons (n=1280), and the second column is for those games played in cold climates (n=114), as defined here. Since many playoff games are played in cold weather, the third column is for all playoff games (n=50+5). (Super Bowl scores are not included because there is no home advantage.)
| Reg. Season | Cold | Playoff | |
| Home | 22.3 | 22.5 | 25.0 |
| Visitor | 19.8 | 18.6 | 19.9 |
The second table looks at scores from the same sets of games differently. The average scores of the winning and losing teams are listed. (Super Bowl scores are included as playoff games here.)
| Reg. Season | Cold | Playoff | |
| Winner | 26.8 | 27.0 | 28.9 |
| Loser | 15.3 | 14.1 | 16.5 |
Cold weather doesn't appear to have a large effect on scoring. It seems to slightly enhance the spread between winner and loser by depressing the score of the loser. This is likely due to the "dome at cold" effect discussed in previous posts.
Playoff scores are generally higher, both in terms of winner and loser, and for the home and visiting teams.
It doesn't appear that cold weather reduces scoring.
I can understand where the perception might come from. Because dome teams are at a disadvantage playing outdoors in cold weather, it follows that they would score less. Many competitive dome teams in recent years have been ones with fast-scoring offenses. The Vikings, Rams, and Colts of recent memories all featured very strong offenses. When these teams were competitive late in the season (and when people were paying attention to them), they would be expected to score less when playing outdoors. But this effect on dome teams would be limited to these specific circumstances and not affect teams in general.
When the Giants played in Green Bay or the Chargers played in Foxboro yesterday, we should not have expected low scores due to the cold temps. Although the frigid sub-zero temperatures yesterday were extreme, even for Green Bay standards, the point is that the weather affects both offense and defense.
Playoff Predictions - Conference Championships
Game probabilities for the NFL conference championships are listed below. Probabilities consider only the last 10 weeks of football. The first half of the season is excluded and recent playoff performance is included, accounting for regular season and playoff strength of schedule.
The probabilities are based on an efficiency win model explained here and here. The model considers offensive and defensive efficiency stats including running, passing, sacks, turnover rates, and penalty rates. Team stats are adjusted for previous opponent strength. Click here for a sortable comparison of playoff team stats from the regular season.
| V Prob | Game | H Prob |
|---|---|---|
| 0.30 |  SD at NE  | 0.70 |
| 0.28 |  NYG at GB  | 0.72 |
Does Cold Weather Change the Game?
Recently I looked at the importance of various phases of the game in winning playoff games. I analyzed and compared regular season games featuring only playoff-caliber opponents and playoff games themselves. This analysis began with the question of whether defense really does win championships, but since has focused on broader comparisons as well.
In the last post, I found that over the past five seasons teams with the better run defense won slightly less than half of games between playoff-caliber opponents. But in the playoffs, the team with the better run defense won 67% of the time. With 114 regular season match-ups between playoff-caliber teams and 55 playoff games during the period studied, the difference may only be due to chance. However, in the comparison of regular season games and playoff games, pass offense, pass defense, and run defense did not show any differences nearly as large as run defense.
[Edit: Based on a 2-sample unpaired t-test, the difference in winning percentage of the better defense (67% vs 48%) is indeed statistically significant at the p=0.02 level. In other words, the sample sizes are large enough to say it is extremely unlikely the difference is by chance.]
Here is the table from the previous post. The winning percentage of the team with the superior stat is indicated. For example, the team with the better offensive passing efficiency won 52.6% of the regular season "good vs. good" match-ups and won 63.6% of playoff games.
| Stat | Good vs Good | Playoffs |
| Home | 59.6 | 64.0 |
| O Run | 45.6 | 45.5 |
| D Run | 48.2 | 67.3 |
| O Pass | 52.6 | 63.6 |
| D Pass | 51.8 | 56.3 |
| O Int Rate | 50.9 | 58.1 |
| D Int Rate | 55.3 | 58.1 |
| O Fum Rate | 55.3 | 40.0 |
| D FFum Rate | 54.4 | 54.5 |
| Pen Rate | 47.3 | 52.7 |
One possible explanation for the difference in the importance of run defense could be the weather. The playoffs are played in January when the weather is cold and often windy in most NFL cities. Teams might bias their play selection toward the run because of the perceived increase in passing difficulty.
To test if weather is the reason for the observed difference in the importance of run defense in the playoffs, I analyzed regular season games played between any-caliber opponents in cold weather. Without direct temperature and wind data for each game, I defined 'cold weather' as being played outdoors in December in a city that averages below 40 degrees wind chill. There were 118 such games between 2002 and 2006.
(I also looked at just the games between playoff-caliber teams played in the cold. However, there were only 12 such games, so the results are not very meaningful. I also expanded the definition of playoff-caliber to 9+ win teams, for which there were 26 games. I'll list both results anyway in case anyone is curious.)
Below is the table of results. Again, the percentage of games won by the team with the superior stat in each category is listed. The first column (Reg. Season) is for all regular season games and all opponent types (n=1280). The second column (In Cold) is for all games played in cold climates (n=118). The third column (9+ Wins) is for games played in cold climates between teams that ultimately finished with 9 or more wins (n=26). The last column (10+ Wins) is for games played in cold climates between teams that finished with at least 10 wins (n=12).
| Stat | Reg. Season | In Cold | 9+ Wins | 10+ Wins |
| Home | 57.4 | 66.0 | 71.8 | 91.0 |
| O Run | 55.0 | 55.4 | 45.8 | 50.0 |
| D Run | 50.0 | 48.8 | 58.3 | 50.0 |
| O Pass | 63.8 | 65.6 | 66.7 | 66.7 |
| D Pass | 59.8 | 60.4 | 45.8 | 58.3 |
| O Int Rate | 59.5 | 61.0 | 54.2 | 33.3 |
| D Int Rate | 59.4 | 59.4 | 33.3 | 33.3 |
| O Fum Rate | 60.8 | 65.0 | 54.2 | 33.3 |
| D FFum Rate | 58.0 | 61.3 | 54.2 | 50.0 |
| Pen Rate | 54.1 | 55.2 | 50.0 | 33.3 |
I'll address the results from the first and second columns. The importance of each stat appears about the same in cold weather as in all games. Other than home field advantage, fumble rates are the only stats that indicate any significant difference in cold weather.
Focusing on run defense, we see that the team with the superior run stopping ability only won 48.8% of the 118 games played in cold weather. This rate is very close to the overall rate of 50.0% for all regular season games. This suggests that it is not the weather but some other factor in the playoffs that may enhance the importance of run defense.
The increased importance of home field advantage in cold weather is probably due to the 'dome at cold' effect, in which dome teams tend to have very little success playing outdoors in cold weather.
Although nothing was conclusively proven with this analysis, there are indications that playoff football is different than regular season football. Although the sample sizes weren't large enough to make solid conclusions regarding most variables, there is enough evidence to suggest that the playoffs comprise a special set of circumstances that may change the dynamics of the game. The level of competition, the weather, the prospect of elimination, or other factors may influence strategies and performances.
At the very least, we can see how fans may perceive defense as being more important in the playoffs. Whether there is truly a systematic link between run defense and playoff success, or it is only the randomness of a small sample, may not be relevant. We have witnessed teams with stronger run defenses win more playoff games. It is apparently, if not in reality, the most important part of the game come January.