For any down and distance situation, a defensive coordinator wants to know how likely a run or pass play will be. He needs to select the right personnel and the right defensive scheme for the situation. Take the 2nd and 10 situation, the second most common down and distance combo in the NFL (1 in 5 of all 1st downs results in a 2nd and 10). In general, offenses tend to pass 55% of the time and run 45%. So defensive coordinators need to be equally prepared for any kind of play type. Or do they?
For offenses to be most effective, they need to be unpredictable. In the 2nd and 10 situation, this means defenses would have to prepare for the nearly equal chance of a run or pass. Many analysts refer to 'balance' as the key to unpredictability. But balance itself doesn’t matter if the offense is predictable in achieving its balance. Running and passing on every other down would provide perfect balance but would be completely predictable. That’s why randomness is at least as important as balance to keeping the defense on its heels. Anything other than random play selection provides a pattern, however subtle, that an opponent can detect and exploit.
In a recent article, I discussed an interesting pattern in NFL 2nd down plays illustrated below. Note how runs are far more common on 2nd and 10 than either 9 or 11 yards to go. The graph is basically continuous and smooth except for a notable spike in run plays on 2nd and 10.
This struck me as odd because 2nd and 10 is not tactically different than 2nd and 9 or 11. The situations are basically the same. My theory was that offenses were running more frequently on 2nd and 10 because that situation arises most often due to an incomplete pass, and offenses tend to predictably alternate between passes and runs. This would result in the unexpectedly high percentage of run plays on 2nd and 10.
I’ve dug into the data now, and I’ve confirmed my suspicion. The graph below illustrates the relative share of pass and run plays based on what kind of play the preceding 1st down was. On 2nd and 10, teams indeed run more often after a pass than after a run, and vice versa.
(The data consist of 14,384 2nd down plays in the 1st through 3rd quarters of all regular season games from 2000-2007. Fourth quarter plays were excluded to remove the possible biases from 'trash time' and running out the clock.) The graphs should be read as follows: The left column are 2nd down plays following a pass, and the right column are plays following a run. The blue portion of each column is the % of pass plays, and the yellow portion is the % of run plays.
This is significant because armed with this information, defensive coordinators can select personnel and plays tilted toward the anticipated play type. They no longer have to be on their heels without an idea of what to expect on 2nd and 10. If the previous play was a run, a coordinator can now be 72% confident the next play will be a pass.
But not all teams have the same tendencies. Compare Brian Billick’s Ravens with Bill Cowher’s Steelers over the 2000-2006 period. The Ravens were far more predictable compared to the Steelers. Cowher’s teams selected 2nd down plays without regard to what kind of play was called on 1st down, but Billick’s teams tended to follow a run with a pass.
(Statistically, the Ravens’ difference in proportions is significant at p<0.001. Typically, a single team’s proportions would need to be within about 8% to be considered non-significant. But that still would not indicate good non-predictable play calling independent of the previous play. The proportions would typically need to be within 3% to be more likely due to randomness than not.)
One counter-argument to this analysis is that teams are wisely choosing to run after an incomplete pass. If a pass falls incomplete, that would be fresh evidence about an offense’s ability to complete passes against this particular opponent. A run stuffed at the line is similarly an indication of each team’s relative strength. Shouldn’t offenses shy away from unsuccessful tactics? Doesn’t it make sense to try the alternate strategy next?
I would say no for two reasons. First, this would be a classic example of the small-sample fallacy, otherwise known as the hasty generalization. Over 40% of all passes are incomplete in the NFL. The outcome of a single pass should not be the basis of a change in strategy, however slight. The sample size of an entire game of passes would still not be enough to make conclusions about its relative merits as a strategy against a particular opponent. Second, even if the evidence of the single trial were so overwhelming, tending heavily toward the alternate play type on successive plays makes the offense too predictable, as we’ve seen here.
Coaches and coordinators are apparently not immune to the small sample fallacy. In addition to the inability to simulate true randomness, I think this helps explain the tendency to alternate. I also think this why the tendency is so easy to spot on the 2nd and 10 situation. It’s the situation that nearly always follows a failure. The impulse to try the alternative, even knowing that a single recent bad outcome is not necessarily representative of overall performance, is very strong.
So recency bias may be playing a role. More recent outcomes loom disproportionately large in our minds than past outcomes. When coaches are weighing how successful various play types have been, they might be subconsciously over-weighting the most recent information—the last play. But regardless of the reasons, coaches are predictable, at least to some degree. Fortunately for offensive coordinators, it seems that most defensive coordinators are not aware of this tendency. If they were, you’d think they would tip off their own offensive counterparts, and we’d see this effect disappear.
In case anyone's interested, here are some other team’s tendencies on 2nd and 10. I picked these teams because they’ve had the same head coaches over the entire period of the data set.
Predictability on 2nd and 10
By
Brian Burke
Subscribe to:
Post Comments (Atom)
Looking at the last 2 charts, and how they make New England and Denver look very perdictable on 2 and 10, does that not make their offenses even better? People know what they are going to do but still cannot stop them (Denver have one of the top ranking offenses over the past 5 years I believe).
That's a fair interpretation, but I don't think anyone is aware of these tendencies. So I doubt defenses could key on them. And NE and DEN tendencies weren't any stronger than the rest of the league.
Perhaps time management is a strong reason for the alternation of play calls.
If you pass incomplete, you've taken about 10 seconds off the clock (the risk of incomplete on 2nd and 10 leaves you with an obvious pass play on 3rd and 10, and the whole series could be a 3 and out taking 30 seconds off the clock). So you run on second to get the clock moving. However, if you ran on 1st down, then the clock is moving, so you can go ahead and try a pass.
some random thoughts:
- you've shown how it works after 'failure', but how about after 'success' (i.e. 2 and 5 or better).
- how many runs for 0 yards are there, really? Of the 14k plays, i would think most
are from pass incompletions.
- so how does predictibilty effect win/loss records? That is the bottom line. Many good teams have signature plays that they will run, even though the D is ready for it.
- The expected probability has to be tempered with the expected gain from the defensive side. Knowing there is a 80% chance of a run, doesn't mean you go go after the run completely, if that means 20% of the time you give up a long pass. Lots of D has a 'bend don't break' philosophy.
- if you do aggressively go against the run, the QB can still audible away based on the defensive formation. If there is 8 in the box, a quick throw can be called (and should be called).
-bob
I wonder how dramatic the effect would be if you looked at what teams do following runs for less than 2 yards or something like that,
quick question about the graphs above, is that not holding constant for second down position
if the average second down following a pass is 2nd and 4 whereas the average second down following a run is 2nd and 6, isn't 2nd and 6 a better passing down than 2nd and 4, according to your graph from before?
what you ought to be graphing are 2nd and 10's following passes and run, 2nd and 9s following passes that gain 1 yard and runs that gain 1 yard, and so on
additional thoughts
wonder if the effect is even stronger if you've had 2 runs in a row, is the third play going to be a pass? etc. etc.
wonder if you've just gotten a 1st down with 2 consecutive runs what the chances of a pass are after that
the possiblity for future study are vast
This post pretty much confirms what Doug Drinen found on his p-f-r blog regarding 2nd-and-10.
http://www.pro-football-reference.com/blog/wordpress/?p=245
He also found that alternating on 2nd-and-10 plays doesn't really help the result. It might be interesting to see whether teams with predictable playcalling tendencies on 2nd-and-10 fare any worse on those plays than those with more random patterns. My guess is that predictability really doesn't matter all that much, but that all teams should be passing more on 2nd-and-10 plays.
Also, you might want to read up on the research that Roland Beech did a couple years ago on playcalling predictability. Some interesting ideas there, too.
http://www.twominutewarning.com/playcall2.htm
I read (several times) the two minute warning article linked above. At first I was excited that it had found some ground breaking stuff.
Essentially, the article assigns each team a predictability ratio that compares how often they run in "passing situations" and vice versa. Through correlations and regression, the article (convincingly) finds that when teams are able to force the opposition to predictably pass in what are normally running situations (presumably by being so good at stopping the run), they win. The top 5 teams at forcing passing running situations in the sample were 60-20, and the bottom 5 teams were 26-54.
Wow, strong evidence for the importance of run defense?
Not really. I realized that teams that are passing a lot in run situations are teams that are already ahead by a lot, and therefore likely to win. It's the same old running/winning fallacy.
It might be more accurate to say teams that are already ahead force their trailing opponents to pass a lot.
Brian,
as a detroit lions fan, I can assure you from experience that you are exactly right. When down by a lot, you pass the ball. Martz got canned in detroit supposedly for not running the ball enough, and if you look at the pass/run ratios and the scores, it is obvious that he was passing cause he was losing by a lot.
-bob
further evidence:
NFL teams have a less than 50% winning percentage (don't recall the exact figure but it is significant) when they have a QB who throws for more than 300 yards. This struck me as counter-intuitive when I first discovered it, but at second thought it makes perfect sense.
See Drew Brees... This leads me to a hypothesis: Gun slingers on crappy teams; are they more valuable fantasy players because they are forced to throw? A good team may really only be looking at 3 quarters eligible for aggressive pass offense. What is the trade off between increased pass-yards and increased risk of the int when playing from behind? Someone want to test this empirically?
Phil's comments suggest that it is not the outcome of 1st down that causes coaches to run more on 2nd and 10, but the likelihood of success on 3rd and whatever you end up with after 2nd down. Obviously if you pass, you risk a significant (40%) chance of facing 3rd and 10, which is a very bad situation to be in. Running gets you to 3rd and 6 or 7, on average, with much less dispersion, which greatly improves the chance of conversion. With your conversion data and a bit of game theory analysis, it should be possible to show if coaches facing 2nd and 10 are treating it as a 2 down series and adjusting their strategies accordingly.
That's interesting. It's like a regular CFL series!
My comment above about the Two-Minute-Warning article should say teams that pass a lot in running situations are already behind and will probably lose, not the other way around. Sorry for the confusion.
I understand it from the coaches perspective. The absolute last thing they want is a 3rd and 10. So they're going to run it to get at least something to help the 3rd down.
As for why there's the "spike" in rush attempts at 2nd and 10 compared to 11 or 12, I guess coaches see anything over 10 as being a loss, and they're going to need two passes just to get the first. In this sense they may be grouping 11 and 12 yard situations in with 14, 15 etc, which is a psychological thing. A loss of even one on first down "feels" a lot different than a no-gainer or incomplete pass.