by Matt Meiselman
Matt has been helped me crunch some numbers this off-season. He is graduate of the University of Maryland with a degree in broadcast journalism. He's originally from New Jersey, but loves New York sports. Matt aspires to work in sports media and has a passion for sports statistics.
We're trying hard not to be broken records on 4th down decisions, but this one is special. -BB
Jason Garrett is not typically regarded as a savvy 4th down decision maker, and his clock management skills aren’t the greatest either. Garrett consistently makes blunders in way too many of his strategic decisions. But even after all the mistakes he’s made as the Cowboys head coach, he somehow found a way to top every one of them with a single mistake in Week 2 against the Chiefs.
With 3:50 to play in the 4th quarter, Dallas trailed Kansas City 17-13. It was the Cowboys’ ball, but they faced a 4th and 10 from the Chiefs 35. They held all three of their timeouts and undoubtedly still had a decent opportunity to come away with a win. Garrett had a decision to make: should he go for it? Punt? Or a kick a field goal?
Based on the 4th down calculator, the Cowboys chose the worst of the three options, and it wasn’t even close.
Based on this chart, Dallas had a 25% chance of winning if they tried to convert the 4th down, a 36% chance of winning if they punted, and an 18% chance of winning by attempting a field goal. The punt may seem like the best choice, and it most likely is, but a case can be made for going for the first down because succeeding would have raised the Cowboys’ win probability all the way up to 49%. It’s nearly impossible to make a case for the field goal attempt, however. Dan Bailey did in fact convert the 53-yarder, raising his team’s win probability to 24%. But as you can see on the chart, that’s still lower than a punt by a significant margin.
It doesn’t take a genius to figure out that scoring 3 points when trailing by 4 late in a game is not ideal, but it may not be entirely clear why the punt is so much better. If Dallas had punted, they likely would have pinned the Chiefs inside their own 15-yard line. Teams playing closer to their own goal line tend to play more conservative, and the Cowboys would have gotten the ball back with better field position after a stop. There’s also the fact that teams needing a touchdown are better at moving the ball down the field than teams needing a field goal, because coaches get way too conservative in the latter situation and hurt their team’s efficiency.
After the Cowboys converted their semi-meaningless field goal, the Chiefs got the ball back with a 17-16 lead. They then picked up a couple first downs before punting it back to Dallas with just 16 seconds left. Tony Romo completed a pass to Demarco Murray on the next play and time expired with the Cowboys never really having a chance to win the game.
The numbers guys that constantly complain about decision-making in the NFL are usually pleading for coaches to be more aggressive. This was one of the rare situations where it would have been much better to be more conservative.
This analysis is correct, but it is funny that it depends on the assumption that all the other coaches are dumb. Basically what you are saying is that it is often better to be down 4 than to be down 1 because you will make better play calls (and the defense will go into the prevent). But maybe Garrett's mistake was just to assume the other team was rational and would play the correct offense from the 15 and the correct defense if they had to punt.
I am no fan of Jason Garrett, but I won’t be too quick to criticize him on this one. First, I don't think the right answer is intuitive. I was leaning toward going for it and frankly, I thought punting was the worst option in real time. I'd be curious to know what others thought before they saw the numbers.
Also, the WPs look a little odd to me. After punting the Cowboys would be down 4, with the ball on the KC ~10-15, and have a 36% WP. With a successful field goal, the Cowboys would be down 1, with the ball on the KC 20, and have a 24% WP. That seems off. If I was guaranteed a successful field goal, I’m not sure I could punt there, even after having seen your numbers.
It's neat to see these 4th down decisions where the numbers say punt when - the mistake always seems to be not going for it. It's a nice reminder that these decisions aren't so obvious.
I know the WP model relies heavily on historical data, which begs the question how it can be adjusted for present day rule changes. For instance, there was an increase in touchbacks last year since teams kicked off further forward. The WP still assumes teams will be returning kicks the way they were before last year, which involves better starting field position for the return team, more return TDs, and perhaps more illegal procedure penalties against the kicking team.
I agree that it could be a stretch to say a punt is better than a made field goal in this situation, but given the low probability of such a long attempt it's pretty obvious that the field goal was by far the worst choice.
I’m not sure about that. If we assume that the WP of a made field goal is 36% (the same as a punt), the WP of a field goal attempt is 24%, essentially the same as going for it.
The WP calculator shows that if a team has a first down with 3:50 left on its own 10-20, it has a higher WP being up 1 than being up 4. So, like Andy said, I think the reason the 4th Down Calculator says punting is correct is because if a team is up 4, they are inclined to play conservatively and give the ball back quickly. But if they are up 1, it becomes obvious that they still have some work to do on offense, and they play more optimally.
Regardless, if punting is truly correct here, it’s a pretty subtle point. Was anyone yelling at their TV for Garrett to punt? I wasn’t. I have to save my Garrett criticism for his more blatant mistakes.
The WP model certainly mathematically favors the punt in this instance. However, as many of us are well aware, sometimes the ball doesn't always "bounce" the right way. Garrett had simply looked at the performance of his team on both sides of the ball throughout the game and mitigated risk based upon his subjective evaluation of what position would allow the team the best option to win.
Clearly, at this point in the game as the final minutes are winding down, the Cowboys had found themselves in the endzone just ONCE - throughout the entire game. Punting in this situation would suggest a certain faith in your offense to overcome this game-long trend and find the endzone for the win. EH! It didn't look like Tony was going to be able to pull that one out of the hat in Arrowhead that Sunday.
The defense had held the Chiefs offense fairly well all game - the only complaint being the lack of TOs. By taking the three points Garrett had (with sound reason) committed to allowing the DEFENSE to achieve a more likely path-to-victory: getting the ball back, (ideally, with manageable time left) needing only a field-goal to win. I mean hell, they already knocked down three, Garrett simply KNEW HIS PERSONNEL and knew what his best chances were.
The averages swallowed within the WP model, IMO do not merit harsh criticism (all the time). I believe these calls made by the HCs can involve a just as much subjective-art as it can objective-analytics.
Oh, and the site could certainly become more entertaining by steering away from the fourth-down emphasis. I mean, "we get it," by now. Coaches are far "too conservative."
You may just as well change your domain to:
"Advanced_Nfl_Fourth_Down_Decision_Making.com (with occasional nonsensical rants by John Morgan)"
I absolutely LOVE your early-articles - and, newer ones as well.
To call it the worst decision of all time seems a tad harsh. Let's put this into perspective. For these stats to be valid, you need to have a rather large sample size. And if you're telling me you have a large sample size, then you're telling me that this decision has been made numerous times, and not only has it been made numerous times before, it has been successful numerous times before.
If you're down 4, you need the ball. The best way to get the ball is to either pin them or convert the 4th down. The worst way to get the ball back is to guarantee the other team decent field position. I wouldn't have expected the punt to be better than going for it but I would've expected the FG to be the worst option
I'm a Cowboys fan and I thought punting might be the best option. When doing the analysis you have to think of the downside too - a miss (which is very likely at 50%) gives the Chiefs the ball at their own 42, which means it's likely that the Cowboys will have terrible field position next time they see the ball, if not down 7 after a Chiefs field goal.
Add in the fact that making the field goal isn't particularly beneficial, and there's definitely a very strong case for punting and trying again next drive.
Isn't there a high probability that if the Cowboys punted then the ball would've wound up in the endzone for a touchback? I don't see any real benefit to a punt in that situation unless you pin your opponent inside the 10, so I wonder what the statistical probability of that is.
If a punt-touchback happens then the Chiefs have the ball on the 20, with the Cowboys needing a stop, then a touchdown, instead of needing a stop and a field goal (after Bailey brought them within 1, and assuming the ensuing kickoff results in a touchback).
It seems to me Garrett made the best choice. He obviously has a lot of confidence in Bailey, who's an outstanding kicker. The defense was stepping up big all day, so the fact that the Chiefs got a couple of first downs was the biggest surprise to me.
I don't trust a set of numbers that don't make sense. As another poster said, they're basically stating that the Cowboys have a better chance of winning by being down 4 points instead of being down 1 (with everything else being more or less equal). I think the dataset is the problem, some of the numbers don't seem to have the proper context.
Jason Garrett realized his offense was most likely not going to score a touchdown in any case. So he had to try for a 2 field goal approach. He trusted his defense could stop KC as it had done for most of the game. Unfortunately, the defense did not hold up in the last drive. If it would have done a 4-and-out, the Cowboys would likely have had the ball at their own 30-35 with about a minute left in the game.
Jack is right, all this changes now that the have moved the kickoffs up, and there is a much higher probability that the receiving team would start at the 20 (or inside it.)
I know these models are based on historical data, but what about the intrinsic knowledge of the strengths and weaknesses your own team? Historical data might show that 1 TD is easier to produce than 2 FGs, but Dallas' woeful redzone conversion rates might suggest otherwise for that particular team.
Can we safely say that Frazier's decision to kick a FG was the worst one of all time? Normally the 4th down calculation amounts to weighing the benefits of making it vs. the downside of failing. But in this case the Vikings had an 87% chance to win if they went for it and FAILED and had a 75% chance to win if they MADE the field goal. So it would have been significantly better for the Vikings to take a knee rather than kick a field goal. Not to mention the chance of scoring a TD if they went for it.
Brian,I'm a big fan of your approach and think that you're onto something but at this point in the development it isn't appropriate to try to apply them to specific game situations. Several posters have alluded to issues translating results from historical data to specific situations. For the current analysis to apply to this situation you must assume that both teams are going to perform near historical averages. This appears to be a questionable assumption. This is a starting point in analyzing coaches decision and is an interesting academic exercise but I would suggest that you qualify this type of analysis with "based on historical averages ... "
it seems to me that over the last 2-3 years FG kickers have become much more accurate. Many 50 yard FG's look like extra points as far as the difficulty goes, and this enhanced accuracy my not be built into historical data. Along with game specific weather conditions and current team play.. (Romo looked injured and could not throw down the field for instance)
I think the cowboys had a much higher chance for a successful FG than historic chances would give them, and a lower chance for a TD in this particular instance than historical averages might suggest.
Also more Kickoff's going for TB's combined with the fact Baily might be accurate up to 60 yards on this particular day changes the needed yardage after a (theoretical) KC punt. I think the decision was not a bad one considering Romo's condition, the kicker in question, current KO rules and outcomes, and the specific in game offense/defense match-up at the time.
The data simply doesn't make sense. While historically accurate, it makes no sense that a team, with under 4 minutes to play in the game and leading by 4, would tell the refs, "Spot them 3 points, please" if given the chance. Nor would their opponents ever say, "Wipe three off the board." in the same situation. To then use that as your basis for calling this one of the worst 4th down calls ever, that's crazy.
So I played with the win probability calculator a little bit:
Up by one, with 3:50 remaining in the 4th quarter, you have the ball with a first-down:
- On your own 10: WP 79%
- On your own 20: WP 76%
- On your own 30: WP 74%
So trusting these numbers, if we are down one and have to punt from midfied, we have to instruct our punter to pin the ball on their 30, rather than on their 10, rigth? Right?
More seriously, while I love Brian's work and writing, he seems to have sense for the limits and weaknesses of his model. If your model says something nonsensical (it's better to be down 1 than down 4), you should recheck your data and your model. (I think Nate Silver would scream "Overfitting!") I just do not believe that the WP calculator isn't even close to being as precise as Brian assumes for his 4th-down writeups.
Sorry, had to edit and re-post.....
Jason Garrett has been fairly criticized at times for his late-game decision making but he unquestionably made the correct decision in the scenario presented in Kansas City: 3:50 4th quarter; 4th and 10 KC 35 yard line; Dal losing 13-17; Dal 3 TOs remaining. Advanced statistics provide a very valuable analytical tool. However, it would be folly for an NFL coach to rely “exclusively” on advanced statistics in making any game decision. Coaching is an art, not a science. A good coach must have an intuitive “feel” for the game, and his personnel, in making decisions, especially in critical situations.
Why would any coach elect to punt from the opponent’s 35 yard line, in non-inclement weather conditions, unless his field goal kicker were injured or otherwise physically incapacitated for some reason? Especially in as game where his team were down by four points against a pretty stout defense that had been playing very well at the time – it was 4th and 10 @ 3:50 4th quarter and the Cowboys had only scored 13 points thus far -- and there is a 50% probability of converting the field goal. And his team has an exceptionally reliable field goal kicker who had nailed several clutch kicks the previous season, as a rookie, and had made a 51 yard FG earlier in the game.
If Jason Garrett had elected to punt, the Cowboys would have needed to score a touchdown to win the game, realizing his team had earlier failed to score a touchdown with first and goal from the five yard line. If the kicker had missed the field goal, the team would have been confronted with the exact scenario. In either situation, their defense would have needed to stop Kansas City on its ensuing possession and their offense, assuming no punt return by their special teams unit, would have needed to drive the length of the field for a touchdown thereafter.
It’s far better to have attempted and made the field goal, as the Cowboys’ did, thereby cutting Kansas City’s lead to one point and given your team a chance to win the game by kicking another field goal, without ever having had to drive the length of the field for a touchdown.
It would have been even more asinine for Jason Garrett to have elected to go for the first down on 4th and 10 in that situation. 4th and 10 is an absolutely dreadful down and distance in the NFL. According to the chart, there was only a 35% probability of conversion. If they had failed to convert, they would have wasted a 50% probability of scoring three points by kicking a field goal. And, since they were in an obvious passing situation, the play might have resulted in an interception or fumble (as had occurred twice earlier in the 2nd half), which could have been returned for a score or at least left Kansas City in good scoring position. Moreover, the game was played in arguably the most hostile arena in the NFL outside of Seattle. It has been reported that the crowd noise level had reached 136 decibels in the second half, which is louder than a jet engine. The players were having difficulty hearing each other inside the huddle, much less at the line of scrimmage. If Jason Garrett had elected to go for it on 4th down, the Cowboys very easily could have gotten a delay of game or off-sides penalty, as they had been prone to do in clutch situations, which would have resulted in 4th and 15 and completely knocked them out of field goal range.
Kicking the field goal was the best decision in the game situation presented. The strategy worked perfectly but for the defense’s inability to contain Jamal Charles on Kansas City’s final drive, as they had done so admirably throughout the game, and Morris Claiborne’s pass interference penalty on 3rd and 10 at Kansas City’s 44 yard line, with less than three minutes remaining and the Cowboys having used all of their timeouts, which effectively enabled Kansas City to run out all but 16 seconds of the remaining game clock.
In response to a lot of the comments calling the data "nonsensical" or claiming that teams would never do something, I think the biggest point these people are missing is that they're assuming optimal behavior from the opposition, which, for the most part, is not true of either historical coaches or coaches currently in the NFL. Now, I haven't looked at the data myself (to see what the sample sizes are, if there any biases, etc), so I'm not saying what each decision definitively should be, but I can offer some plausible explanations for why the data may say something that appears counterintuitive. For example, being up 3 in many cases at the end of the game can be better than being up 6, since while the opposition only needs a field goal if they're down 3 (in other words, they have less field to cover), they're also much more likely to be content with the 3 points, so the downside is oftentimes just overtime, which still means a 50% win probability. In a second example (or specifically, the Chiefs-Cowboys example), a team that is up 4 with the ball is much more likely to play conservatively (i.e. run 3 straight times) than a team that is up 1 with the ball, as they are likely to be deceived into believing that the 4 point lead is a lot more secure than it actually is. And to the anon poster a couple of comments above who gave the example of being up 1 with 3:50 left, assuming that the example is correct, two possible explanations are 1) that the offense has more urgency to get the first down with worse field position and 2) that assuming the offense doesn't get a first down, they're more likely to get the ball back with enough time to make a drive of their own after the other team's field goal since the other team will often be content with gaining 20 or so yards into "field goal range." Obviously, one has to make adjustments from the historical numbers based on personnel (most important of which are the quarterbacks and the opposing coaches), and given that most non-Belichick non-Harbaugh coaches make decisions based on what has traditionally been done, there isn't even much to adjust there. In this example, Dallas has Tony Romo (an above-average quarterback at the very least, so the need for a TD down 4 is less of an obstacle than average) and KC has Andy Reid (who I would guess behaves very traditionally in terms of late game decision making, so there's likely to be a significant difference in how conservatively he plays up 4 vs. up 1), so I see no reason to suggest that kicking a field goal is a much better option than what the historical numbers say.
Too much faith put in models. You will never convince me giving the Chiefs the ball on their 15 down 4 is more likely to win than giving the Chiefs the ball on their 20 down 1. It does not compute. Kicking the FG was the right call, models be danged.
George, is there any situation in which you, having the ball with less than 4 minutes in the game, up by 4, would rather be up by 1? If a magic power gave you the option of wiping three points off the board immediately, is there *any* situation you would do it? Likewise, as a defense, down by one, under four to play, would you tell the genie, "Give them three more points, they'll play worse."?
If not, then this simply doesn't make sense. I don't disagree with the historical data, bu some historical data can't be used in a predictive context, it only works in a historical one. This is one of those times.
Going for it on 4th and 10 and not getting it would mean the game is over. These numbers don't mean anything to me. While watching the game, Dallas was banking on stopping Kansas City on their first set of downs. Kansas City had trouble moving the ball all day and couldn't run the ball much. Dallas' D-Line was winning the battle ALL GAME! If it wasn't for that pass-interference, which WAS pass-interference, Dallas would of gotten the ball back with plenty of time to get into Field Goal range. Punting would have been the worst option, and going for it probably wasn't too smart either, considering the Chiefs defense was tough all day. The Field Goal was the right choice, but they just couldn't hold Kansas City.
To phrase my objection (I am the same "anonymous" who wrote "nonsensical" above) a little more factually:
I just do not believe that the historical NFL data is large enough of a sample size to accurately determine the win probability as a function of three variables: score difference, time left, field position. (And that's ignoring down and yards to go, of course.)
To convince me otherwise, maybe Brian can let us know how many games there are with a score difference of 1 late in the 4th quarter, and then use some estimates to give error terms for his win probability calculator (which is now estimating a function of two variables based on this sample).
Garrett had no GOOD option to choose from in that situation, so to call it "the worst move any coach has made on 4th down this season" somewhat misses the point IMO.
The real issue with the play-calling is that Garrett got his team in a lose-lose situation. 1st and 10 on the KC 35, down 4 with 4 minutes remaining, looks like a pretty decent situation.
The one thing you absolutely want to avoid here is the 4th and long, practically taking away the option to go4it on 4th down and forcing you into the 50+ FG.
Of course, all of the "percentages" are based on experimental probability. All decisions would preferably be made to increase your expected value (EV), but you must first analyze the data and understand what variables are affecting calculations.
With that being said, I'll give my thoughts on the decision without any reference to the "statistics." As I watched the situation unfold, I had many thoughts. The ball is at their 35, it's a low percentage kick, our offense has stunk all day, we're down by 4, and our defense has been very good. Regardless of any decision and the outcome, two things must happen: (1) Our defense must get us the ball back, and (2) We must score again.
I understand Garret's decision, but I would have chosen to punt (without any reference to EV). In no way do you want to give the ball back at the 35 by trying a low percentage field goal when you have to score again anyway. We're going to have a very long field in that scenario if missed, and even somewhat of a long field if the FG is made because, in both situations, we have the PUNTER FROM HELL on their team!
Try to pin them with a good punt, they will go conservative with a 4 pt. lead, and now we hopefully have a chance to get the ball back with "not horrible" field position and 4 downs to play with instead of 3. It's the "percentages" of marching down a very long field that would have determined my choice. Nonetheless, this is definitely NOT the worst decision (if a bad one at all) that I think has been made by any coach.
To all those criticising the model I would say that, whilst it seems counter-intuitive that a 1pt lead is better than a 4pt lead, the fact remains that in that precise situation teams with a 4pt lead lose more often, hence the percentages the model produces.
The reasons behind this would almost certainly need further investigation but, assuming there are no errors in the data, the historical results are what they are - facts.
This does not mean that the decision is necessarily a poor one and I would agree with those that say Garret needed to take into account his knowledge of personnel on both sides and some sort of feel as to how likely a TD or second FG would have been. But the original post states that the gap between the options is 18% - meaning he has to be sure that his 'knowledge' gives an 18% swing in his favour by making the decision he did. Otherwise it's still the wrong choice, no matter how counter-intuitive the 'right' choice may seem.
Can you rerun the probabilities, but move the Cowboys from the 35 to the 30, and 25, and 20, and 15? i.e. how far forward would the Cowboys need to be before a FG *was* the right choice.
I suspect the impact of a 4-point lead vs 1-point lead on your analysis is (could be?) off.
There's no point on the field where it would be the right decision to kick a field goal down 4 with 3:50 left on 4th and 10 (according to the model). At some point it becomes more advantageous to try to convert a first down which actually isn't until the 15 yard line. However, a team that's losing benefits more from using high variance strategies so the spot where it becomes more valuable to go for it is subjectively more like the 30 yard line.
This proves yall all to smart for your own good because none of you even noticed that Jason Garrett is not a punter, so he couldn't have even punted if he wanted to.
I used to play special teams and offense and defense
Is there a way to look deeper into the distance to go variable? I believe the NFL calls anything up to and including 36 inches to go, 1 yard, and anything >36 inches but <= 72 inches is 2 yards, etc. But I have to believe there is a pretty big difference between 2 inches to go and 35 inches to go (probably less difference between 108 inches to go and 144 inches to go). Is there any way to look at that?
Of course, I forgot to add that a practical consideration to the actual distance is that any "4th and 1" data will include all occurrences at 1 inch and at 35 inches, but if the outcomes between those two cases are significantly different, then lumping all the "4th and 1" data together can create problems.