The Falcons' 4th Down in OT

With 10:52 left in overtime against the Saints, the Falcons faced a 4th and inches at their own 29. Head coach Mike Smith decided to for it. Was it a smart call?

Calculating Win Probability (WP) in OT is surprisingly simple compared to regulation time. Except for the final few minutes when there is real possibility of a tie, time is not a factor and the score is always tied.

A punt would be the conventional call. A typical punt from the 29 nets 38 yards, giving the Saints a first down at their own 33, worth 0.58 WP (a 58% chance of winning). This makes intuitive sense, because teams that win the coin toss are in a similar situation and win just under 60% of the time. The Falcons would therefore have a 0.42 WP following a punt.

If the Falcons successfully convert the first down, they'd have a fresh set of downs at (at least) their own 30, good for a 0.57 WP. Notice that if they succeed the situation is nearly the symmetrical opposite of the punt. Atlanta would have the ball at slightly worse field position as New Orleans would following a punt.

If the Falcons fail to convert, they'd hand the ball over to the Saints at their own 29, a nearly fatal situation, worth only 0.18 WP. I realize this is surprisingly high, but it reflects actual overtime game results. Often, coaches will conservatively run three times into a brick wall to set up a 40-plus-yard field goal, which is far from a slam-dunk.

Conversions on 4th and 1 are typically successful 74% of the time. But this includes all 4th "and 1" situations, everything from a yard-and-a-half to go to an inch to go. For now, let's stipulate that it's 74%.

The total WP for the conversion attempt is:

0.74 * 0.57 + (1 - 0.74) * 0.18 = 0.47

The go for it option is worth, on net, a 0.47 WP. That's better than the 0.42 WP of the punt option, at least according to league-average percentages. I'm certain the fact that Drew Brees and the rest of the Saints offense is not league-average helped sway Mike Smith's mental calculus.

If you do the math the other way around, the break-even conversion probability would need to be 62% for the conversion attempt to be worthwhile. And if we don't buy the 0.18 WP following a failed conversion attempt, it would still have been worth going for it. Even If failing to convert meant an instant loss, the possibility of success would be slightly better than punting (0.74 * 0.57 WP = 0.43 WP).

Smart call. It just didn't work out. To Sean Payton's credit, he got aggressive and didn't settle for the long field goal attempt. The Saints moved the ball into chip-shot range and won the game.

76 Responses to “The Falcons' 4th Down in OT”

1. Anonymous says:

Agreed, I thought the exact same thing. But all the headlines called it stupid and foolish.

2. John Candido says:

Great article settling what most are wondering this afternoon? Real question though is, what was the specific difference in the conversion probability of handing it off to Michael Turner in that situation compared to having Matt Ryan QB sneak it? I think that was the egregious mistake that the 4th down call to go for it will get the heat for.

3. ShroffishBoil says:

Have you ever done the same analysis for the 4th and 1 by Switzer in 1995?

4. Wheell says:

Some notes: Due to changing NFL conditions the success rate for a run on 4th and short is lower than the historical average. Furthermore kickers have become more effective at 40+ yard field goals. However, while those 2 effects make the decision relatively worse, it is unclear how whether or not they make it 5% worse in terms of win%.

http://sports.espn.go.com/nfl/columns/story?columnist=pasquarelli_len&id=3112032

Seth Burn

5. Anonymous says:

0.18 WP simply cannot be right. "Right" meaning what one should estimate in the future. I would assume the sample of OT games is too small, or that FG accuracy has increased sufficiently over the time span of your database such that your odds of making a long FG are too low relative to current expectations.

Let's assume the Saints run three poor running plays in a row and gain a total of only 6 yards. So they try a 40-yard field goal on 4th down. Teams are 77% kicking FGs of 40-49 and 88% on FGs of 30-39 this season (278 total attempts). So let's say a 40 yarder is 82%. Under these assumptions, the odds of the Saints winning on that posession alone would be at least 82%. Of course, if they miss, they don;t lose. They merely give up the ball at the 30, and have a 0.43 WP. So, 0.82*1 + 0.18*0.43 = 0.90 WP in that situation. I think 0.90 is slightly conservative if anything, and certainly more realistic than 0.82, given the current skill level of NFL kickers.

6. Anonymous says:

u know what is interesting, before that NFL screwed around with the kickoff many years ago teams losing the coin toss in overtime actually won more than the coin flip winner

now with the kickoff now the same as many years ago, teams losing the coin flip is actually 5 and 1,

maybe baseball isn't the only sport with a shaky comissioner

7. The Wizard says:

anonymous (at least one of them :) )makes a very good point. I think the .18 has to be high and the big factor as he stated is that not making a field goal, you are still close to 50% to win the game.

I know Brian won't want to do this, but if all of us mention the stupid articles that Chris Chase writes on yahoo(he called Smiths deicsion foolish) on this site, maybe he will get wind of it, and stop writing such stupid articles.

As i was watching the game, I had no doubt that Smith made the right decision to go for it.

I hated the play. Has anybody ever done a study on fourth down conversions by trying a qb sneak with less than 3 inches? Not sure how avaialable the data is, but over many years, I would think it happens enough to gain a decent sample size? Also, with QB sneaks, it seems that the officials can not see through the big pile are going to error on the side of the offense for a few inches, which of course can be the difference. I would guess, and it is just a guess from watching many of those situations, that a QB sneak from 3 inches or less picks up a first down 75%-80% of the time and handing the ball off to ANYBODY will make it at most 67% of the time.

8. Brian Burke says:

Yes. Moving the kick off line does more to equalize OT than those silly rules for the playoffs the league instituted last year.

Also 0.18 WP *is correct*. It's not just a sliver of a sample at exactly the 29 yd line. It's a smoothed model across all field positions.

I'll post the raw data in a moment.

9. Brian Burke says:

I wondered why they didn't go with the sneak too.

Here's the raw data:

Row Labels Average of win
1-5 1.00
6-10 1.00
11-15 0.90
16-20 0.91
21-25 0.90
26-30 0.84
31-35 0.82
36-40 0.79
41-45 0.77
46-50 0.84
51-55 0.66
56-60 0.68
61-65 0.58
66-70 0.68
71-75 0.48
76-80 0.56
81-85 0.45
86-90 0.45
91-95 0.37
96-100 0.20

10. Frank says:

Would there be a way to do an article on QB sneak vs traditional run vs PA on 3rd/4th and short? The obvious issue is that QB sneaks become more likely as the value of the "1" becomes shorter, which might be tough to account for. Would still be interesting though imo - as a reasonably analytically-minded fan, my impression is that sneaks are far undercalled and runs straight ahead far overcalled recently in these situations.

11. King Yao says:

But what happened to Mike Smith's personal equity? I'll bet it went down the second he decided to go for it on 4th down. If it is successful, his status likely doesn't change much. Since it was unsuccessful, the probability of him getting fired and blamed for the Falcons not making the playoffs (if they indeed do not), goes way way up.

12. King Yao says:

Belichick's personal equity didn't change when he decided to go for it on 4th down at IND a couple years ago, but I can't think of another coach that has as much job security as Belichick does.

13. Anonymous says:

Brian:

This article is why readers should love your site. This will be the call that everyone is talking about tonight and tomorrow and you provide the readers with the actual numbers to analyze Coach Smith's decision. At worse, it is a very defensible decision and Smith certainly doesn't deserve the ridicule that he is receiving for his decision. If he punts, the most likely decision is that NO drives 20-30 yards and kicks the same game winning field goal. Thanks for the good work. Jason

14. Anonymous says:

What about Tom Coughlins decision to go for it in OT deep in the Giants own territory? It was 4th and 7 or 6 at around the 30 if I recall correctly.

15. Anonymous says:

Also, to add, note no one talking about that decision since it was successful. Hard being a head coach in the NFL.

16. Josh Katz says:

Anon: Agreed. Although it wasn't OT, but with the Giants trailing by 7 with 3:33 to go in the 4th. Giants' ball, 4th and 6 on the NYG35.

Troy Aikman was going on about how he disagreed with the decision. Meanwhile, the break-even success rate to make going for it the right call is 8%.

17. Josh Katz says:

(And the average success rate is 45%.)

18. Anonymous says:

Percentages?Are you kidding me?Your team plays 4+ Qts of football and you bet the whole game on one play.Not so bright.

19. Brian Burke says:

Put yourself in the shoes of a NO fan.

You see your team make a stop on 3rd down, and the punt team starts to trot out onto the field. You'd think, "Sweet."

Then all of a sudden the offense goes back out onto the field, and you think what? I know what I'd think, "Oh crap."

That should tell you something.

20. Jack Moore says:

I would argue if you don't feel comfortable enough to bet a game on picking up two freaking inches your team isn't very good.

21. Michael Beuoy says:

"Your team plays 4+ Qts of football and you bet the whole game on one play.Not so bright."

And there you have the perfect expression of the "lose slowly" strategy.

Although it does really surprise me that teams with the ball 11-15 yards from the endzone still manage to lose in OT 10% of the time.

22. Scott Kacsmar says:

Atlanta had almost the exact same situation on the first drive of OT. Instead of 4th & 1 at their own 29, they had 4th & 2 at their own 28. They punted, and the Saints started at their own 27 (they went 3 & out). I would have done the same thing again. Just not convinced a conversion there for a 1st & 10 at your own 30 is worth the risk of not getting it. If it was more at midfield? Sure. If there were only a few minutes left in the game? Yeah. But this one didn't feel right and I don't know why the QB sneak isn't more automatic in these situations.

Coughlin's call on 4th down was arguably more interesting. You don't get that, might as well say game over given how good Akers has been this season.

23. Jonathan says:

"Also 0.18 WP *is correct*. It's not just a sliver of a sample at exactly the 29 yd line. It's a smoothed model across all field positions."

Is the data strictly from overtime games, or does it include data from the regular season? If it's the former, then I have to admit that's surprising & very interesting.

24. Rum says:

I think the real question here is, as other people have pointed out, why call a straight up run instead of the QB sneak?

Also, similar to the QB sneak which attempts to go under the pile, why don't running backs in situations of 4th and an inch try to go over the pile more often by leaping?

I have to think both the QB sneak and the "RB leap" have a better shot at converting than the straight run into the pile.

25. Ian Simcox says:

Or, Rum, why not pass? The field is 50 yards wide, why call a straight up run between the hash marks that restricts the field to barely over 6 yards wide?

Look at the video. There are 10 Saints players within 3 yards of the LOS. All the Falcons are there, so it's going to be 9 blockers v 10 defenders and one very congested area of the field.

Spread it out. The way to convert a 4th and 1 is to treat it as a normal play. Use the whole field. There were 70 yards of perfectly good turf beyond the LOS, why restrict yourself to using only one of them?

26. Anonymous says:

I agree with Ian here. I'm surprised we don't see more bubble screens to a big WR/TE. High percentage play where a bigger player can take advantage of a smaller player, but with fewer extra defenders to complicate things.

27. Phil says:

This is why this site is great. As soon as Smith made the call, I thought to myself "can't wait to see this analyzed on advancednflstats.com". Thank you.

28. Anonymous says:

"Just not convinced a conversion there for a 1st & 10 at your own 30 is worth the risk of not getting it"

It is. Someone did the math.

29. Brian Burke says:

Jonathan- The WP numbers come from just OT periods. The conversion rates and punt distances are from the proverbial 'normal football situations'.

30. TOP says:

The best part was, one of the talking heads (I think I heard on national radio) was saying how Smith wasn't playing aggressively to win! The punt was the "aggressive" move to that guy.

31. Anonymous says:

As others have said, this discussion reminds me why I come to this website. Even Peter King today references Brian's work and numbers and then STILL comes to the conclusion that it was a better decision to punt because basically that's they way he feels. Ouch. How many cognitive or mental errors are in his "analysis". Anyway as others have said, I loved the decision to go for it, but I HATED the playcall. I was with my Dad and we yelled "slow developing play" and sure enough it got stuffed. My favorite for 4th and inches is to have a spread formation with the QB under center running a sneak.

32. Anonymous says:

Just want to 2nd (or 3rd or 4th...) the previous commenters that as soon as I saw the play I said to myself, "Well, that'll be on Advanced NFL Stats in the morning." Thanks as always for the great work Brian.

33. Ian B says:

Ian S., according to the below article by Brian in 2008, the optimum mix would call for a pass at most 20% of the time in short yardage. Passes haven't been as successful as runs on 3rd and 1. I think that Brian uses 3rd down conversion rates for 4th down plays so I would guess that the same mix holds.

This weekend the Giants had a 4th and 2 at the SF 10 and needed a touchdown to win the game. Eli attempted a short pass over the middle and Justin Smith batted it down. JS said that the coaches instructed the entire D-line to put their hands up and that it was blind luck that he got a hand on it. There are always things that can go wrong on "sure fire" plays.

34. comet52 says:

A gamble is only the right call when it works. When it fails you look like the Hindenburg.

This play call was the equivalent to saying you're all-in in a poker game. Is that necessary when there are 10 minutes left in o.t.? What % advantage do you need to justify a gamble like that in a game where you still have plenty of time for a big play, a favorable bounce, a defensive stop, etc. to happen?

The idea that the calculated edge (assuming it's correct - it's based on history which is never a perfect predictor of present events) justifies gambling for the outcome at that point is questionable to me.

35. Mark B says:

Brian,

I love your site and analysis. I thought and think going for it on 4th and inches from that position on the field is the right call, and I write that as a dyed in the wool Saints fan.

I do have two questions / comments:
I find WP / WPA very interesting and useful but wonder about implied risk aversion. Does it operate under the constraint of risk neutrality? Would be interested to see the impact of different risk constants, perhaps incorporating the work of Kahnemann and Tversky. Risk-loving as well as risk-aversion is not necessarily irrational; too often I think discussions around decisions like this one seem to come down to either you’re rational and risk neutral, a maniac and risk-loving (Les Miles, Andy Reid), or a job-preserving Neanderthal and risk-averse. Under what risk-tolerance curve would a coach be indifferent between punting and going for it under ATL’s circumstances Given the distance was only inches, one presumes that risk aversion is infinite. Maybe losing slowly is what fans really want, for it provides ample rationalizations and those give fans comfort their team is “above average.” As Jeff Goldblum’s character in “The Big Chill” said, “Try to get through the day without one good rationalization.”
Re the QB sneak, I could see it attempted out of the spread off an LOS audible depending on the DL formation / alignment, but out of a heavy formation I don’t think it’s cake. The defense’s first priority is to take away the A gaps. DTs line up in those gaps and are coached to angle-shoot the QB’s lower legs, the idea being to pin him or set him up for the LBs / DBs to clean up. These A gap shoots also take away the QB “wedge.” At any rate, the OC is pitted against both DTs if the QB sneak is called: one vs two. There’s no real blocking scheme for a QB sneak when the defense is double A-gapped. At least when you run the B or C gaps you can actually apply a blocking scheme, concentrate blocking at the point of attack, use motion, etc.

Anyway, keep up the great work.

Mark B

36. James says:

I agree with Ian too. Spread it out and make the defense decide what it wants to defend:

- if the linebackers crowd the line run a WR screen or slant.
- no safeties and/or man coverage? Hitch and go for the win (a la Cowboys-49ers OT game).
- line up under center and if the linebackers play back QB sneak.

Basically anything other than announcing to the defense exactly what you're going to do, let them get the best personnel and playcall, then run it right at them.

37. Scott Kacsmar says:

"It is. Someone did the math."

An interpretation of it, sure. You have to ask yourself how much faith you're willing to put in averages, because I don't think this game represented the average situation very well. Good teams, good offenses, one terrible run defense, one bowling ball of a RB, and in a dome. And isn't Atlanta not far removed from some record season in terms of yardage allowed via punt returns? Same punter too right?

The whole 0.47 vs. 0.42 hinges on the average punt starting the Saints at their own 33. I think it would be more convincing to post a range of starting FP after the punt and the change that has on WP.

This game actually saw the Falcons punt from this area on the field 3 different times. Let's look at what the WP would have been for ATL each time.

1st quarter: Falcons had 4th & 1 at ATL 29, punted 52 yards, returned for 2, penalty on NO pushed them back to 10.

ATL WP (w/penalty): 0.65
ATL WP (w/o penalty): 0.51

As you'll see, penalties can always happen on either team on a punt. With or without it, Atlanta's in better shape than the 0.47 from going for it.

2nd quarter: Atlanta had 4th & 1 at ATL 29 and punted 35 yards for no return. ATL was flagged 15 yards for fair catch interference, putting NO at the NO 48.

ATL WP (w/penalty): 0.31
ATL WP (w/o penalty): 0.39

With or without the penalty, the Saints were in a better position this time.

OT: ATL had 4th & 2 and punted from the ATL 28. NO got it at their own 27 after no return.

ATL WP: 0.46

This WP is all but equal to the go for it option (0.47).

It's worth noting that in all 3 cases, the Saints went 3 and out with the ball. In OT, if you felt confident in starting the Saints at their 27 or worse, it would give you the same or better WP as you'd have going for it on 4th & 1 at your own 29.

And to think this could have all been avoided if Cox caught the ball right and got the first down on the previous play.

38. Anonymous says:

"You have to ask yourself how much faith you're willing to put in averages"

At least enough for the media not to be second guessing you to death. You can trust the numbers, plug in your own variables instead, or go with your feelings. But no one can say Smith's choice wasn't rational, or that the risk wasn't worth it, I think.

39. Kevin says:

TMQ addressed this issue of how to run short yardage plays in last weeks column.

http://espn.go.com/espn/page2/story/_/id/7204143/tmq-says-offensive-creativity-trickled-short-yardage-plays

comet52 said...

"A gamble is only the right call when it works. When it fails you look like the Hindenburg.

This play call was the equivalent to saying you're all-in in a poker game. Is that necessary when there are 10 minutes left in o.t.? What % advantage do you need to justify a gamble like that in a game where you still have plenty of time for a big play, a favorable bounce, a defensive stop, etc. to happen?"

The problem was not that the Falcons went all-in by going for it, but that they went all-in and showed the Saints their hand by coming out in their heavy formation which declared to the defense what play they were probably going to run. The Saints only had to defend a small part of the field. If you spread the field by going 3 WR, then the whole field has to be defended and the conversion is more likely to be successful.

40. Anonymous says:

"The idea that the calculated edge (assuming it's correct - it's based on history which is never a perfect predictor of present events) justifies gambling for the outcome at that point is questionable to me"

Your focus on time baffles me. Overtime is sudden death. If the Saints march down the field after the punt and score, it's over. Now, that might not happen, just like you might not convert the fourth down attempt. But you're gambling either way.

This notion everyone has that going for it is gambling while punting is not also baffles me. Especially considering, at least according to one analysis--and probably many others--punting is the long shot.

41. Anonymous says:

"punting is the long shot"

Or, rather, the marginally longer shot.

42. Anonymous says:

"This play call was the equivalent to saying you're all-in in a poker game."

Here, precisely, is where most people go wrong criticizing the decision, and why most NFL analysts are misguided (well, this and the whole "doesn't trust his defense" BS). They are deceived by the All or Nothing drama of the situation.

I've heard again and again the supposed gap between upside and downside. Fail to convert and you pretty much lose the game then and there; convert and you still have to march all the way down the field. a shot at winning similar to what you'd have if you force a punt anyway. Well, three things:

1) although it gives the Saints great odds, failing to convert is not a death sentence;
2) you'd have to march down the field after forcing a punt or perhaps getting a miracle turnover anyway;
and 3) you have better odds to win if you convert than if you punt.

It's a gamble either way, though the alternate option may not be so "all in." But sometimes it makes sense to go all in, which is why poker players do it so often.

43. Anonymous says:

I have a question. Aren't WPs calculated based on games in which most coaches play according to "conventional wisdom?"

In other words, isn't the real definition of the WP "this is the chance that you have of winning the game in OT if you have the ball at this position on the field and play according to the rules that most coaches in the past have played by?"

And since those coaches almost always punt on 4th and 1 from the 29, doesn't that invalidate using those winning percentages in order to calculate the likelihood of succeeding in a situation in which you go for it on 4th and 1 from your 29?

I mean how many of the OT games used to create the WP data actually feature coaches going for it on 4th and 1 deep in their own territory? I would imagine only a handful. Thus, an assumption already built in the data you're using to calculate whether or not to go for it is that you're operating in a world in which coaches don't go for it on 4th and 1 from their own 29.

44. Boston Chris says:

Ah, the poker analogy. Often brought out by the commenter with little understanding of either poker or statistics as applied to football.

45. Scott Kacsmar says:

"At least enough for the media not to be second guessing you to death. You can trust the numbers, plug in your own variables instead, or go with your feelings. But no one can say Smith's choice wasn't rational, or that the risk wasn't worth it, I think."

It's not a question of it being rational. It's questioning whether or not it was the right call based on one interpretation of the numbers.

Smith decided on a rational risk, and it did not work. Those are the facts. Saying it was the right call or better decision, that's still up for debate.

46. stevekirsch says:

Two points:

1. I like my chances of picking up an inch better than stopping Brees and the Saints offense short of a FG.

2. These models are built on historical averages, which may or may not be representative of the current situation. All situations are not the same and are not made in a vacuum. Sure, if you played it out 1000 times you'd come out ahead, but you don't. You have once chance to do or die. This is one huge way that that the NFL differs from MLB in that there aren't 162 games for the averages to play out. For example, a rookie RB and a beat up OL vs. a healthy Ravens D. This could potentially sway the probability model significantly. I think Brian accounts for this well in his analysis, but this is one of those "feel" decisions that don't go over well with these crowds. Coaches can't make moronic decisions, but given personnel situations and the error in these models there are tons of other variables that equate to a coaches feel. I think making decisions based solely on historical data would be a bad practice.

The great MLB pitchers don't throw a curveball because historical data across MLB shows the probability of a swing and miss on 2-2 after a fastball inside is 0.08 better than his slider. It because he has a feel for the individual hitter's timing, their tendencies, reactions to other pitches in the AB, his feel for his own pitch effectiveness that day, etc. The same factors go into a coach's 4th down call. So I think there are significant factors in the "noise" that we can't really account for. This model is the best we have, but its not the be all end all solution and may not be as useful as we think.

47. Boston Chris says:

..."Coaches can't make moronic decisions, but given personnel situations and the error in these models there are tons of other variables that equate to a coaches feel. I think making decisions based solely on historical data would be a bad practice."...

Okay, let's apply your logic. M. Smith went for it, so we should rely on the "coaches (sic) feel" and assume it was the right decision. Why should the MMQB with less information than the coach presume he is right, while the coach is wrong?

48. Jonathan says:

I'm just surprised that teams manage to lost the game 18% of the time after taking over on the 28 yard line.

"The great MLB pitchers don't throw a curveball because historical data across MLB shows the probability of a swing and miss on 2-2 after a fastball inside is 0.08 better than his slider."

Not really an accurate comparison. A pitcher relies heavily on being unpredictable, and is more comparable to "which playcall should I use."

Conversely, the defense knows going in if you're going for it or now. They don't see Matt Ryan line up under center and then get fooled into sending on the punt return team.

49. Scott Kacsmar says:

I'm 99.999% sure the 18% is not for "taking over" on the 28 yard line. It's for teams that reach the 28 yard line with a 1st & 10. They could have started there, or they could have started at their own 25 and moved the ball to get there.

50. Jonathan says:

Scott, this is true, but I think it's reasonable* to assume that "taking over on the 28" is the same thing as "driving 50 yards to get there," from a win probability point of view.

51. Anonymous says:

The model is useful in retrospect, but the real time decision is based on the coach's belief in his offense. Does anyone really believe that an NFL offense shouldn't be able to make a few inches with the proper play call? If you don't think (know) that your offense can gain a couple of inches, then you need to fire the OC, the offensive line coach, and everyone else associated with the offense. If you want to win the game, the decision to go for it is a no-brainer.

The discussion should be about the play call--not the decision to go for it. Looking at the offensive and defensive formations I thought any thing other than QB sneak was dead. I don't have any real data, but it seems that heavy offensive formations against goal line defenses often result in lost yardage. QB sneaks seem to at least regain the line of scrimmage--and in a game of inches that's probably enough.

Les

52. Anonymous says:

Probably should expand a bit on comment about reaching the line of scrimmage. As pointed out by the Wizard, if the QB at least gets to the LS, by the time the refs sort out the pile up, they'll probably spot the ball a few inches forward, and that's all Atlanta needed.
Les

53. sunrise089 says:

Most of the comments above are great, but I love the few comments here and loads of them in the media repeating the silly "stats compiled from thousands of plays are bad, my gut tells me Atlanta should have punted, I back up my gut up with these stats from a sample size of New Orleans' last three drives." ;)

54. Ed Feng says:

A pretty close call by the numbers. However, let's also mention that New Orleans is last in rush defense (5.2 yards per attempt). That should play a role in the decision making.

http://thepowerrank.com/

55. Memphian says:

Seems right to me - I was traveling and used 60%/60%/20% as my numbers instead of 58/58/18.

Too bad the media are too dumb to understand it and only look at the downside.

56. Jim Glass says:

It's not a question of it being rational. It's questioning whether or not it was the right call based on one interpretation of the numbers.

Smith decided on a rational risk, and it did not work. Those are the facts. Saying it was the right call or better decision, that's still up for debate.

The fact that "it did not work" is irrelvant to whether it was the right call. This must be determined as of the time the decision is made, with only the information available then.

One find's one's home is bet on one roll of the two dice, and must pick the number to bet on. The right call is to bet on 7, period. That's the number that has the best chance of winning. The issue ends there.

Will it win? Probably not. If it doesn't, does that make it the wrong call? No, it was the right call. Suppose 7 loses and snake eyes wins, does that make snake eyes the right call? No. To choose a 1/36 chance of winning as the right call and a 6/36 chance as wrong is deranged.

The awfully, awfully large number of football fans and pundits who conclude that since a play didn't work it was the wrong call (did work so it was the right call) make this mistake all the time.

Do analysis of the play without knowing the result, which can only bias what one thinks. If one must know the result, don't think about it. Ignore it. Even refering to the result while discussing the analysis indicates the bias at work.

57. Jim Glass says:

A gamble is only the right call when it works.

False, see my prior comment. You illustrate it well.

When it fails you look like the Hindenburg.

What you "look like" is irrelevant as to whether it was the right call. People used to "look smart" by saying the Earth sits at the center of the universe.

The idea that the calculated edge ... justifies gambling for the outcome at that point is questionable to me.

If you must make a choice between A and B, then *whatever* you choose is a gamble. There is no way around it. You can't say choosing heads on a coin flip is a gamble but choosing tails isn't.

Your use of the word indicates you think it means "increase the chance of losing". But that is wrong. "Go for it", was a gamble. "Punt" was *equally* a gamble. Either choice was a gamble. The question is which was the better gamble -- which produced the highest probability of winning.

The actual outcome of the game is irrelevant to that.

58. Scott Kacsmar says:

I don't see why there has to be a "right" call, especially when the numbers show anything but an obvious decision. This isn't a black & white issue.

If you want to do the math and show that one option was in fact rational. That's fine. That's been done. That's indisputable.

But using that number to say that it was beyond any doubt the right call, well you're not going to sell me on that. Not when I can clearly see a couple of yards on the punt changes the answer.

59. Jeff Clarke says:

"It's a gamble either way, though the alternate option may not be so "all in." But sometimes it makes sense to go all in, which is why poker players do it so often. "

AMEN!!!

There are quite a few situations in poker where it makes perfect sense to go all-in. When the circumstances dictate make your move.

60. Mike says:

"This play call was the equivalent to saying you're all-in in a poker game."

Assuming you're in a cash game and playing within your bankroll (i.e., there are no other outside factors to consider), if you aren't willing to push all your chips in the center with a 50.1% edge, you're playing the wrong game.

There's merit in mentioning that coaches face pressures re: job security, facing the media, etc. However, when you excuse a poor (or, nicely, suboptimal) decision because of those reasons, you're only perpetuating the culture that demands that one punt in this sort of situation because "Well what if you miss?"

61. bigmouth says:

One question the occurred to me as I was trying to explain this to someone: don't we need to do a similar expected value analysis for the possibilities associated with a punt (e.g., Saints return the kick for a TD, muff the punt, etc.)?

I'm not sure we can just assume the WP associated with a kick is the same as the Falcons' WP if the Saints get the ball on their own 33 yard line.

62. Brian Burke says:

bigmouth-That's a good point. I've looked at that. The chances of significant returns a blocked kick or fumble are really low, and to a large degree offset each other. It doesn't affect the results enough to be worthwhile.

63. Anonymous says:

not sure where the WP is coming from, but it would seem to me that IF I were the coach I would look at the following.

Likelihood or success * likelihood of getting another 40 yards * likelihood of FG success

VS

Likelihood of failure * likelihood team stays in position to attempt FG from failure point * likelihood of FG success

ex. success .7 * .5 * .7 * +3 (points) = +.88
failure .3 * .? * .7 * -3 (points) = -.??

It would seem the likelihood of the gaining 40 yards even after gaining a fresh set of downs is much lower than a team not losing any yards, possibly making the expected value negative. I'd also argue this is all about using historical data that is really irrelevant and what you should be basing your decision on is that fact that you just basically had 2 consecutive 3 and out series (including this one), so you in all likelihood will not be kicking a FG with this decision.
My point, is as soon as Smith decided to go for it I knew the game would be over IF Atlanta failed to convert. The oppositions WP (however calculated) under that circumstance would seem to be very high especially since Payten isn't likely to pound the ball 3 straight times. I just saw it as old school, ATL had forced a 3 and out previously, make them march it down the field to beat you. Likelihood of that given the previous 3 and out?

64. Jason says:

Take a look at the ESPN Power Rankings (http://espn.go.com/nfl/powerrankings/_/year/2011/week/11)

"Statistical analysis showed coach Mike Smith made the percentage play with that controversial fourth-and-1 decision Sunday. (Sando)"

Wonder where he got that analysis from?

65. JMM says:

Is there a standard deviation or error bar associated with the .42 and the .47? Are they the same number or are they different?

66. Bigmouth's Slate Buddy says:

Are you factoring in weather / the dome as it pertains to kicking into your calculations?

67. Bigmouth's Slate Buddy says:

Also, if you punt, you likely have more plays with which to fail, but still recover and win. How does that factor into the calculation? As a coach, I'd want more opportunities to out-think the opponent.

68. Jim Glass says:

if you punt, you likely have more plays ... As a coach, I'd want more opportunities to out-think the opponent.

If you're thinking like this you should probably be concerned about giving the other coach more plays to out-think you. That's not being snarky (even if that is a main purpose of the Internet).

Seriously. With 60-odd players and how many coaches contributing to the game, how much is you outhinking someone else likely to suddenly become decisive? Being that it has gained 0 advantage during the first 60+ minutes. And if it does come down to the HCes' competitive thinking, what grounds do you have to believe you are more likely to out-think the other guy rather than he will out-think you (now that Herm Edwards is out of the league)?

Believing one's own brainpower is key to winning a game like this certianly indicates a kind of cognitive bias -- especially when passing on the best chance to win one's got, just to get the extra chance to think.

I used to be a tournament chess player -- where one's own thinking rather than the performances of 60 other people really matters -- and if the other guy stumbled into a position where using some book line instead of thinking gave a favorable probability of winning one would go for the favorable probability rather than thinking every time. Because the other guy can always match thinking or even better-than-match it, but he can't do anything about being stuck with bad odds.

It's interesting how many people when faced with a choice between (a) deciding a game immediately with a 40% chance of winning, and (b) extending the game indefinitely with a 30% chance of winning, evidently will choose (b) because with the extra time ... ????

69. James says:

Anon doing the equations, you said "I'd also argue this is all about using historical data that is really irrelevant".

I disagree, but fine. No historical data.

But then you say, "What you should be basing your decision on is you had 2 consecutive 3 and out series (including this one), so you in all likelihood will not be kicking a FG with this decision."

That's historical data! You are basing your decision on what happened in the past! The only difference between Win Probability and your data is that Brian used years' worth of information and you used two drives.

You might say that those two drives are more relevant (and you're probably right) but that doesn't mean the other information has no value. It's just that the other information includes so much more information than just two drives that the relative importance of those two drives is very small overall.

70. Ian Simcox says:

Right, well I just got through a good session of efficiency adjustments to the win probability baselines and have concluded that going for it was by far the best option.

Of the four efficiency stats, the biggest driver 4th and 1 success is defense rushing yards allowed. Coming into this game, the Saints allowed a league worst 5.3 YPC, which results in a 0.75 chance of converting the 4th down.

The biggest drivers of wins in overtime are field position and defense passing yards allowed (offense passing yards are almost of no consequence whatsoever, oddly). Plugging the numbers into the regression, punting gives the Saints a 0.71WP, a successful conversion gives the Falcons 0.59WP and a failure gives 0.09WP.

Bang these altogether, and adjusting for the teams, you get punting worth 0.29WP and going for it worth 0.46WP.

Seeing as before the Saints final drive, Brees was going at 7.4 Y/Att, I think adjusting the NO likelihood of winning up seems fair.

71. Anonymous says:

Of the four efficiency stats, the biggest driver 4th and 1 success is defense rushing yards allowed. Coming into this game, the Saints allowed a league worst 5.3 YPC, which results in a 0.75 chance of converting the 4th down.

Is this really valid? Especially with the package they had brought in vs. the package the Falcons had?

Especially with the play call going directly at the heart of the Saints DL in Cody and Franklin who both did an excellent job at getting push against Blalock and McClure (350 and 317 vs. 326 and 296 pounds) with Smith doing an excellent job beating his man, backup LT Svitek, to the spot, and for some unknown reason, backup C/RG? Hawley, trying to pull but failing with Ellis ready to pounce in case Hawley should succeed.

I'd say that the play call is a significant portion of the decision, since you very rarely get into this situation in any game where the offense AND the defense sell out so hard against that 1 yard. The ~75% success rate takes into all situations when not only the possibility of making the 4th and 1 is in play, but the rest of the field is in play (and you potentially don't have 5 DL on the field).

On this play, the rest of the field was significantly NOT in play, and it would not surprise me if the Saints with their beefy DL were actually very good at sell-out goal line situations.

Doing a spot check, I only see 1 rushing TD allowed by the Saints all year of 1 yard or less, by Kuhn, who was not actually the featured back on the play that he scored (that would be Starks) and did not seemed to be keyed on (the Saints were 9 wide on the DL with only 2 backers vs. 7 wide on the DL with 4 backers vs. the Falcons).

But again, I don't know how many attempts have been against the Saints in that type of situation....

72. Ian Simcox says:

I don't see why it shouldn't be valid. After all, its the same defense that allowed that many yards previously and in 4th and 1 situations before, defenses that allowed that many yards allowed the conversion more often. That said, if anyone can find a way to show that NO had the ability to stop a crucial 4th down despite how they've played previously then I'm open to being corrected.

73. Ian Simcox says:

Just had a look at the Saints so far this year in 'X and 1' situations (not a perfect comparison to this, but it's something).

New Orleans has allowed conversions on 10 of 13 'and-1' plays where the play call was a rush, and of the three 'and goal from the 1' plays, allowed a touchdown on 2 of them.

Not that any of this is particularly statistically significant, but it does show that they aren't exactly a brick wall when it comes to stopping a team with one yard to go.

74. Anonymous says:

Again, the issue isn't the Saints' prevention of a conversion on an 'X and 1' situation. It's selling out on one play to stop that 1 yard which is what they decided to do for that 4th and <1.

The equivalent that I can see are the "X and goal from the 1" plays. I see the Kuhn score, which again was a different defensive arrangement which keyed on Starks (and maybe worried a little about Rodgers's bootleg) as they lined up 9 players on the LOS. What's the other score you're talking about?

And considering the key for the defense seemed to be on Turner, it would have been interesting to see what would have happened on say a wide toss (though given that Turner isn't very foot fast, maybe that's why the Saints keyed on stuffing everything up the middle).

The problem, as I see it, is that we're using an overly broad brush when we're saying "oh, of course, they're 75% likely to make it on 4th and 1" which doesn't take the personnel, context of the situation, and plays available to both sides when figuring out the likelihood of success. Now, unfortunately, that's a lot of what we have available, but just because we only have the broad brush, doesn't mean that the percentages we derive from that tool are things we should take as perfect or set in stone or even infallible.

I guess, just as I dislike statements like "Smith is obviously wrong to go for it," I also don't like statements like "Smith was obviously right" when we're using data and measurements which only give us generalized information which we have to make best guesses from.

For example, your talk about how the best driver of 4th and 1 is defense rushing yards allowed.

But against the vast majority of rushes, any defense is NOT going to be playing the same as in a goal line situation. In the vast majority of rushes, they're going to be worried about the potential of being burned on a deep pass, or losing contain and getting gashed on a cutback, etc.

However, I COULD see a correlation (and remember correlation does NOT equal causation) between teams which normally allow a lot of rushing yards and teams which give up X and 1 situations because, for the MOST part, teams who allow a lot of rushing yards are likely to be the ones which don't have the size to stand up to the OL of the other team.

That does not quite seem to be the case with the Saints, particularly vs. the Falcons, which again, speaks to that we might be painting with an overly broad brush here.

Don't get me wrong, I like seeing the statistics and hypotheses of trying to predict what happens. I just think we might be misusing these tools and thinking they're the cure-all and be-all when they really are imperfect.

75. ecthompson md says:

Brian – I'm sorry it didn't write earlier. Been kind of busy. I continue to love your website.

In my opinion, here's the problem with your calculations. What are the chances of the New Orleans Saints stopping the Atlanta Falcons in a fourth in one situation when both teams are in their goal line formations. Atlanta was in an extremely tight run formation. If I'm not mistaken, they didn't even have a wide receiver on the field. Therefore, New Orleans could play the run 100% (I guess they could have thrown it to Tony Gonzalez or one of the running backs out of the backfield.) I really think that the formation that the Atlanta Falcons was in really changes the equation.

As usual, I love your stuff. Keep up the great work.

76. DrZin says:

You don't risk near-certain defeat to gain a first down on your own 30. Ridiculous.

It doesn't require any statistics to determine that Mike Smith's was a jack*ss call.