The Wall Street Journal's Michael Saffino wrote an article last week about the Falcons' aggressive tendencies on fourth down this season. In opponent territory, with 3 yards or less to go and more than 4 minutes remaining, the Falcons went for the conversion on 72% of their opportunities, the most in the league. They converted on 85% of those attempts. The review of the game's play-by-play indicated that Atlanta would have scored 21 points according to league-average field goal rates, but following their successful conversions the Falcons actually went on to score 51 points--a net of 30 additional points. And in 3 games, the points scored off of 4th down conversions made the difference in the win. Could this be the explanation for the Falcons' dramatic over-performance this season?
In this post, I'll attempt a few things. First, I'll discuss how we should properly measure the impact of 4th down decisions and measure the wisdom of the decisions themselves. Then I'll look specifically at the Falcons' 4th down situations and measure their impact on Atlanta's fortunes in terms of EPA and WPA. Lastly, I'll look at how all the teams did in terms of 4th down decisions.
Measuring the value of 4th downs
The wisdom of 4th down decisions must be judged based on the situation and information available at the time. The outcome itself should not be part of the equation when we decide whether a particular decision was correct or unwise. Additionally, it's not quite correct to count up the points scored on the drive subsequent to a fourth down conversion and then say that they made the ultimate difference in the score of the game. Had the Falcons kicked a FG instead of going for it in any of these situations, the entire game from that point forward changes. Each teams' risk/reward equation is affected and their strategies would change. We can't inject or remove points from a game and pronounce the outcome would have been just so. Things have to be measured probabilistically, and this is where Win Probability (WP) comes in especially handy. In fairness to the author, Mr. Saffino only suggests that the points may have made the difference in the end, which is the best one can do without the benefit of a WP model.
Measuring the actual impact of a 4th down conversion attempt is straightforward, at least in retrospect. We can measure the Win Probability Added (WPA) of the play or the Expected Points Added (EPA) of the play. In other words, the impact of any 4th down attempt can be measured by subtracting the WP of the game state prior to the 4th down play from the WP of the game state following the conversion attempt. The same concept applies with EPA.
Measuring the wisdom of the decision itself, before we know the outcome of the attempt, is more difficult. The best way to measure 4th down decisions is to compare the expected WP of a punt or field goal attempt with the total expected WP of the go-for-it gamble. To the degree that the total go-for-it WP exceeds the punt/kick WP, a coach forfeits that amount of WP when he punts or kicks. This how I have typically analyzed 4th down decisions, including this one in which the Falcons made a big mistake.
To summarize, there are two separate measures here. One is a retrospective measure of the impact of the 4th down plays actually attempted. The other measure is the value of the decision itself, analyzed prior to the outcome of the play.
The 2010 Falcons on 4th down
Looking specifically at the 2010 Falcons, it's true that they led the league in attempts under the specifications in the WSJ article--on opponent territory, 3 or fewer yards to go, and more than 4 minutes left in the game. (I think these are fair restrictions because they identify situations in which the conventional wisdom says it is somewhat reasonable to go for it. The 4-minute restriction is intended to filter out situations in which it's typical for trailing teams to go for it.)
The results of these plays added 0.90 WPA. In other words, in total, Atlanta's 4th down attempts created about one additional win in 2010. That may sound fairly impressive, especially for a league in which the difference between 10-6 and 9-7 can be enormous. However, all teams go for it on 4th downs sometimes, at least in some obvious situations, and the league-average WPA on 4th down under the conditions specified in the WSJ article is 0.20, making the Falcon's net advantage 0.70 WPA. That's impressive and should be cheered loudly, but it's not the 3 wins implied in the article.
In terms of EPA, the Falcons' 4th downs translated into 30.7 expected points (over and above FG attempts or punts), which is very close to the 30 net points estimated in the article.
What if we remove the restrictions of opponent territory, 3 or less yards to go, and more than 4 minutes to play? The Falcons 4th down conversion attempts brought them 0.93 WPA, while the league average was 0.52. That's a net of 0.49 WPA for the Falcons attributed to the outcomes of all 4th downs conversion attempts.
Impressive, but this is based on the outcomes, and Atlanta was very fortunate on their conversion attempts. What if we evaluate their 4th downs decisions, based on expected conversion success rates, prior to knowledge of the outcomes? Using the specifications of the article, when the Falcons punted or kicked in situations where the WP for a conversion attempt exceeded that of the punt or kick, they passed up only 0.09 WPA. This ties them for 2nd best in the league. But when we expand the data to consider all 4th downs, the Falcons forfeit 0.75 WPA when unwisely kicking or punting, which puts them square in the middle of the pack at 17th in the league.
All 32 teams
Now let's take a look at how all the teams fared in 2010 on 4th down. I'll give you two tables. The first limits the data to the specifications from the article, and the second is for all 4th downs.
Each table has several columns. The first data column lists the percentage of opportunities that each team attempted to convert 4th downs in situations in which the go-for-it WP exceeded both the kick and punt WP, an indication of how frequently each team went for the first down when it would have been to their advantage. The second column lists the actual net WP as a result of 4th down conversion attempts, and indication of how much each team actually benefited from its 4th down attempts. The third column lists the total WP forfeited when each team passed up a conversion attempt opportunity that would have been beneficial. The smaller the amount of WP forfeited, the more astute the team's coaching staff.
Team | Go4It Rate (%) | Net WP Gained | WP Forfeited |
HST | 63 | 0.05 | 0.07 |
DEN | 70 | 0.32 | 0.09 |
ATL | 71 | 0.93 | 0.09 |
MIN | 50 | 0.01 | 0.11 |
TEN | 43 | 0.18 | 0.11 |
ARZ | 58 | 0.18 | 0.12 |
KC | 71 | 0.30 | 0.13 |
CAR | 45 | 0.14 | 0.13 |
NE | 55 | 0.44 | 0.15 |
SF | 55 | 0.07 | 0.15 |
SL | 45 | 0.30 | 0.16 |
OAK | 63 | 0.57 | 0.17 |
DAL | 50 | 0.36 | 0.17 |
SD | 27 | 0.04 | 0.17 |
NO | 33 | 0.10 | 0.21 |
NYG | 20 | 0.07 | 0.21 |
BLT | 25 | 0.17 | 0.22 |
PHI | 47 | 0.20 | 0.24 |
CHI | 25 | -0.16 | 0.26 |
WAS | 38 | 0.29 | 0.26 |
CLV | 29 | -0.08 | 0.27 |
GB | 31 | -0.10 | 0.30 |
DET | 43 | 0.25 | 0.30 |
BUF | 50 | 0.28 | 0.33 |
IND | 29 | 0.09 | 0.34 |
PIT | 18 | 0.10 | 0.34 |
CIN | 37 | 0.32 | 0.34 |
JAX | 50 | 0.57 | 0.34 |
TB | 24 | 0.15 | 0.39 |
SEA | 35 | -0.02 | 0.40 |
NYJ | 20 | 0.26 | 0.45 |
MIA | 17 | 0.15 | 0.61 |
Average | 41 | 0.20 | 0.24 |
Team | Go4It Rate (%) | Net WP Gained | WP Forfeited |
NE | 21 | -0.01 | 0.39 |
SD | 11 | -0.09 | 0.41 |
SL | 23 | 1.16 | 0.47 |
MIN | 18 | 0.42 | 0.56 |
CAR | 29 | 0.58 | 0.57 |
NYG | 24 | 0.55 | 0.61 |
DEN | 25 | 0.47 | 0.64 |
NO | 21 | 0.06 | 0.64 |
OAK | 30 | 1.54 | 0.65 |
HST | 28 | 1.08 | 0.67 |
BUF | 29 | 0.69 | 0.69 |
JAX | 15 | 0.50 | 0.71 |
GB | 41 | 0.85 | 0.72 |
PHI | 21 | 0.32 | 0.72 |
BLT | 24 | 0.42 | 0.73 |
PIT | 33 | 0.71 | 0.74 |
ATL | 27 | 0.21 | 0.75 |
IND | 22 | 0.95 | 0.77 |
SEA | 21 | 0.90 | 0.78 |
SF | 23 | 0.64 | 0.84 |
TEN | 12 | -0.04 | 0.86 |
WAS | 17 | 0.04 | 0.90 |
DAL | 35 | 0.73 | 0.91 |
ARZ | 28 | 0.42 | 0.93 |
KC | 8 | 0.04 | 0.93 |
CHI | 19 | 0.04 | 0.94 |
TB | 18 | 0.86 | 0.97 |
CIN | 27 | 0.45 | 1.10 |
DET | 37 | 0.76 | 1.14 |
NYJ | 22 | 0.17 | 1.17 |
CLV | 24 | 0.27 | 1.21 |
MIA | 20 | 0.93 | 1.28 |
Average | 24 | 0.52 | 0.79 |
For the most part, teams with small amounts of forfeited WP are the teams with the fewer number of opportunities for conversion attempts. I could present the average amount of WP forfeited for each team, but I don't think that tells us much. The total WP forfeited numbers are still useful because they tells us which teams, and which kinds of teams, would have benefited the most from a more aggressive strategy.
It's interesting that less than a third of the WP forfeited was on plays inside the constraint of opponent territory, 3 or fewer yards to go, and more than 4 minutes left. Part of the reason is that it very often makes sense to teams ahead to go for it on 4th down within 4 minutes. Just like in the Patriots-Colts game from last season, a relatively easy 4th down conversion that seals the game is a better gamble than punting to a team that has all 4 downs to use and time to score the go-ahead points..
Another consideration is that it very often makes sense to go for it when there is far more than 3 yards to go. In no-man's-land, where a FG is an iffy proposition and a punt buys you very little field position, it can make sense to go for it on 4th and long depending on the score and time.
As always, the usual caveats apply. These numbers are based on league-average baselines. One other caveat is that these WP numbers are minimums. They assume a conversion success would get the exact number of yards required and no more. Realistically, they would often gain more yards than needed, making the conversion attempt even more lucrative.
And we all thought Miami was unlucky. Turns out risk aversion played a big part in their disappointing season.
What’s really encouraging for me is that the teams with the highest go for it percentages are actually coming out with the most net WPA. It is what we’d expect, indeed it’s why we’ve been saying coaches should go for it more often, but we all know there’s a huge difference between something working in theory and it actually working in practice. These numbers provide great practical evidence that the more you go for it when the stats community says you should, the more you will win.
One in-game situation where Norv comes out looking good.
Scratch that. Another in-game situation where Norv doesn't look good. I'm guessing the WP forfeited is misleading because the Chargers didn't have many opportunities and the 11% Go4It is indicative of Norv's inability to be aggressive when it matters. Not to mention he should be more likely to go for it because of their offense.
So what does this say about Belichick and his aversion to Go4It despite the Patriots' offensive prowess?
Brian,
Were there any cases this year of teams going for it when your models suggest that they shouldn't?
Brian,
Another mind bending article.
As stated:
"One other caveat is that these WP numbers are minimums. They assume a conversion success would get the exact number of yards required and no more. Realistically, they would often gain more yards than needed, making the conversion attempt even more lucrative."
Can we assume that statement applies to just WP Forfeited? And if so, could it not be accounted for with average yards over needed(or similar)?
Are the "all 4th downs" exclusive of the last 3 minutes of the game and just include the entire field and down and distance or do they include the entire 60 minutes of the game?
Also I know that the article used the last 3 minutes of the game as their cutoff but I think even more interesting might be something like what is each team's go4it rate when their WP% is greater than say 10% and they are starting to get desperate.
Random oddity - KC has been talked about how often they "go for it" and I have seen them go for it a lot myself. Indeed they are #1 at going for it in opponents territory (essentially tied with ATL) but they are dead last at 8% in total go for it at 8%. That seems kind of weird. They must almost have some rule that says if its 4th down at this point of the field go for it and every other option dont?
Anyway very interesting.
No. The 'all 4th downs' includes the entire game, including OT.
Noticed that about KC too. I think you're right. Haley must have a rule of thumb he sticks to pretty tightly.
brian-
very insightful. i find it really interesting how your analysis casts singletary in a much better light re "agreesiveness on 4th down" than much of the fan base would. i'm guessing it's probably because his passiveness on downs 1-3 overshadowed his aggressiveness on down 4, you know. then again, even though there are a lot more "observations" of passiveness in downs 1-3, it might not overshadow agressiveness over fewer observations on down 4 because 4th down plays are much more high-profile than those on earlier downs (except for maybe 3rd down). or, on the 3rd hand, the dichotomy here might not be real, with it just being one of those "see what you wan't to see" situations.
i think i'm going to write something up for NN based on what you've found, so just a question or two for clarification purposes:
1) do i understand it correctly that your analysis only included 4th down plays (either constrained sample or full sample) in which the team was supposed to go for it according to WPA? in other words, your analysis doesn't include 4th down plays where the team wasn't supposed to go for it in the first place, right? i'm just clarifying here, but if not, i think a future post looking at coaches with respect to the converse of this situation might be equally interesting. as in, "how much WP did this idiot coach gain or lose by going for it when he shouldn't have?"
2) you mentioned that the teams with the highest WP were generally those that had more opportunities to go for it. did you run any correlations on this or on anything else related to the associations between the stats in the tables? i'd be interested to know if teams that are higher in go4it ratego are higher or lower in net WP gained and/or net WP forfeited. just wondering if there's a practice effect here.
look forward to your response.
Danny-1) Yes, you're correct. 2) No, I just noticed the teams with the most opportunities were the same teams with the most WP forfeited, and vice versa. It's not something I looked at, but I doubt there is a practice effect. WP is so sensitive to situational variables that they tend to swamp other smaller factors.
To me, the most interesting column is the "WP forfeited" column. No surprise to see New England's famously aggressive Belichick on the top of the heap.
I'd like to see that turned into a rate stat. Maybe take that WP forfeited, and divide it by the number of plays involved in that WP forfeited, PLUS the number of times they actually went for it. It would be interesting to see if such a rate stat could account for some additional variance in the prediction model.
I meant only that the points gained by going for it equaled or exceeded the winning margin in three games. And it should have said that I assumed no field goals over 50 yards were attempted (not made). But this research is excellent, as always, Brian. (I always wish I had more than 270 or so words for those WSJ Counts.)
Don't get me wrong, Mr. Salfino. That was the best 270 words written about football I've read in while. Keep it up.
Great analysis! It makes me a little sick to see the Jets may have left a win on the table due to poor decision making. I knew from watching that they don't have a good grasp on 4th down strategy, but the size of the effect is surprising.