In Decision Theory, there are generally two kinds of analysis. Descriptive analysis is what people actually do, and prescriptive analysis is what people should do. Rarely are the two things the same. For example, when I use the win probability model to evaluate 4th down decisions, I'm doing prescriptive analysis. Trying to explain whatever the heck coaches are actually doing would be descriptive analysis.

To be fair, coaches are not computers. They are subject to all the imperfections of human decision making. In this post, I'll examine some of the ways that coaches may be making decisions, including minimax, minimax-regret, prospect theory, and expected utility. I'll also discuss the potential for how much of a difference a pure prescriptive analysis can make when applied in real games.

NFL Orthodoxy

NFL football has evolved as extremely conservative game. By that I mean that coaches adhere to the wisdom passed down from previous generations and are reluctant to deviate from the established orthodoxy. In the real world, away from sports, this approach usually makes sense. Unlike sports, the world is not bounded by sidelines, end zones, and 15-minute quarters. It is highly uncertain and far less predictable than we'd like to think. It makes sense to adhere to what is known to work rather than try to engineer an optimized outcome in a highly uncertain environment.

But in football, we have the stats. We know the probabilities. And we know the possible consequences. 'Conservative,' as I defined it, is therefore often not the best approach. I think the reason that so many coaches adhere to the same orthodoxy, whether in terms of playbooks or 4th down doctrine, is because they aren't conscious of the level of certainty available to them.

Minimax

One of the more conservative approaches is the minimax criterion. Minimax says pick the option that assures you the highest minimum utility. Let's say you have the choice between going on a picnic and going bowling. You'd really rather go on the picnic, but it might rain. Your payoff matrix would look like this:

No Rain | Rain | |

Picnic | 4 | 0 |

Bowling | 1 | 1 |

If it doesn't rain, the picnic pays off, but if it rains you've lost the afternoon. Bowling is not as much fun as the picnic, but it wouldn't matter if it rains. Minimax says go bowling because 1 is its minimum payoff while 0 is the minimum payoff for the picnic.

Minimax-Regret

Another decision method is known as the minimax-regret criterion. This method seeks to minimize potential regrets. Imagine coming out of the bowling alley and being greeted by a sunny blue sky. 'Darn. Should have gone on the picnic.' In this case, if you go bowling and it doesn't rain, you've gained 1 unit of utility but lost out on 4 units, for a net regret of 3. If you go on the picnic and it does rain, you've gained 0 utility but lost out on 1 unit, for a net regret of 1. If you want to minimize your regret, you'd choose the picnic.

Notice that I haven't mentioned the weather forecast yet. These methods are best relied upon when there is a very high level of uncertainty in the "states of nature" that will determine the payoffs.

Now consider a football example. Say a coach has three plays that make sense for a given situation, and the opposing defense can call one of three kinds of defenses. An example payoff matrix might look something like this:

Def X | Def Y | Def Z | |

Play A | -4 | 4 | 12 |

Play B | -2 | 3 | 8 |

Play C | 3 | 2 | 1 |

Note that this is not game theory. We're not looking for a Nash equilibrium. The offensive coordinator is thinking of the defense as a "state of nature." It's something he has no control over and is difficult to predict.

In this case, both Plays A and B have the possibility of negative payoffs. Play C guarantees at least a payoff of 1, and therefore would be the minimax decision.

The regret method says something different. Assume the defense had called Def X. The best payoff possible given Def X would be 3 with Play C, so had we called Play C there would be no regret. But had we called Play B, we would have earned a -2 payoff, which equates to a regret of -5. In other words, we could have had 3, but we got -2. And had we called Play A, we would have earned a -4, which is a regret of -7.

If we repeat the regret calculation for each possible defense, we get a whole new regret matrix:

Def X | Def Y | Def Z | |

Play A | -7 | 0 | 0 |

Play B | -5 | -2 | -4 |

Play C | 0 | -2 | -11 |

Given this regret matrix, the minimax-regret criterion would look for the choice that assures us of the best worst-case scenario. For Play A, the worst regret is -7. For Play B, it is -5. And for Play C, it's -11. Therefore, we'd pick Play B because it is the least costly in terms of maximum possible regret.

Of course, coaches or anyone else would never actually draw up a matrix and do the math to make a decision. But just like in the picnic-bowling example, our brains are attempting poor analog versions of these kinds of decision criteria, and emotions play a large role.

Expected Utility

What if we reduce the uncertainty in the defense? We can't predict exactly which one we'll see, but we can estimate the probabilities that we can expect each defense. The expected utility of a choice is the weighted average of the possible payoffs. For simplicity, say each defense is equally likely with a 1 in 3 chance. Now we can estimate the expected utility for each play choice. In the example above, the expected utility for Play A is (1/3)(-4) + (1/3)(4) + (1/3)(12) = 4. The expected utility for Play B is 3, and for Play C it's 2. The expected utility method therefore says Play A is the best choice.

The three methods each call for a different decision. Each method is logical and consistent in its own way, but there is only one truly correct method in football, only one prescriptive analysis. Remember, in football we can know the probabilities and the payoffs, or at least have a solid league-wide baseline for them. The expected utility method is the only correct method.

The math behind expect utility analysis couldn't be any easier. It's 5th grade arithmetic. The challenge is knowing the utility function. Yards, and even points, don't equate to utility. A 7-yard gain is usually good, but it's relatively useless on 3rd and 8. And a 3-point field goal doesn't help late in the 4th quarter when down by 7.

Fortunately, there is win probability (WP). WP is the one and only correct utility function for any game, including football. Winning is all that matters, whether by 1 point or 100 points. WP is also perfectly linear, which is essential to valid expected utility analysis. A 0.40 WP is exactly twice as good as a 0.20 WP, and 0.80 WP is twice as good as 0.40 WP.

Prospect Theory

But even if coaches were to somehow use expected WP analysis when making decisions (say by using 'quick reference' cards like they sometimes do for 2-point conversion decisions), it's likely they still wouldn't be very rational.

Prospect theory says that people fear losses more than they value equivalent gains. Humans evolved with a tendency to try to avoid loss. We're usually more upset with ourselves when we misplace a $20 bill than we are happy when one falls out of the laundry. This tendency has been borne out time and time again in clinical experiments and other studies.

In football, this means that decisions are warped because coaches would fear a loss in WP more than an equivalent gain in WP. The chart below illustrates this concept. According to prospect theory, the "joy" from a 0.05 gain in WP is less than the "pain" from a 0.05 loss in WP.

This asymmetry would affect tactical decisions in many ways, but the most obvious may be 4th down doctrine. Say a team finds itself in a situation where punting would result in a 0.50 WP, but the expected utility analysis says going for the conversion would result in a net 0.55 WP. If the goal is to win the game, the correct decision in this case is to go for it. Period.

The analysis isn't so straightforward for the coach (even if he could do all the math on the spot). Say the failed conversion results in a 0.45 WP and the successful conversion results in a 0.65 WP. A 50% chance at successful 4th down conversion therefore results in a net 0.55 WP.

But the coach sees the 0.45 WP as a possible loss of 0.05 WP, and he sees the 0.65 as a gain of 0.15 WP. Because he fears the loss far more than he values the potential gain, even one 3 times as large, he'll prefer the sure-thing option and punt.

Further, it's possible to actually measure the risk aversion of coaches by comparing the WP advantages in situations where they went for the conversion to the WP advatanges in situations where they forego the conversion attempt.

An Advantage

The coach who can resist this human tendency and make decisions based purely on expected utility will have an advantage. Just how big an advantage, no one can ever know. Actually, that's not true--I'll tell you right now. Just by following a pure expected utility analysis on 4th down, a coach would win an average of an extra 1.4 games per year.

I calculated this based on a play-by-play database from the past 9 seaons. For each 4th down in which a team kicked either a FG attempt or punt, I calculated the difference between going for it and kicking. Wherever the difference was positive, I summed the increase in WP for going for it. The grand total for nearly 2400 games was +203.1 WP, which equates to an increase of 0.17 WP for every game. But since there are always two teams competing in every game, this means that we need to halve that, which is 0.086. The bottom line is that a pure expected utility approach to 4th down decisions would increase a team's chances of winning a game from 0.50 WP to about 0.59 WP. This is equivalent to an extra 1.4 wins per season (0.086*16).

That's a bold claim, I realize. But if you trust my WP model, which is really nothing more than a smoothed empirical observation of how often teams actually won in given game situations in real NFL games, then the claim is not so bold. It's not a perfect model, but the errors are unbiased, meaning it overestimates as much as it underestimates.

Still, if a coach only followed the expected utility recommendations when the WP for going for it was greater than 0.05 more than the WP for kicking, his team would still benefit by an extra 0.8 wins per season. That's nothing to sneeze at in a 16-game season.

Interesting. If every coach started to trust the WP model, would the winning percentage drop back to 50% essentially negating the effect? Is the 1.4 wins only added if you are the only coach (per game) follows this method?

Yeah, the effect is only there if you're the only one moving - if all the coaches had used the WP method to make decisions, they'd evenly spread out the gained WP and leave us back at 0.

But that doesn't mean you shouldn't do it: if 31 coaches use WP and one guy is making old fashioned decisions, he'll lose more.

To use an extreme example, you gain a big advantage if you're the only team in the league that utilizes the forward pass. If all the other teams start using it too, it's now a wash, however, that doesn't mean you'd get an advantage by giving it up.

Right. Like anything else, once everyone does it the advantage is lost. But until everyone else catches up, it could be quite an advantage.

Question: A while ago, during Spygate, you said that the Patriots have somewhere in the ballpark of an un-accounted-for 2 wins per season based on their offensive and defensive passing and running efficiencies. It's also known that they go for it on 4th down more than most people.

You mentioned this in that article, but at the time it seemed you hadn't calculated yet that you could get 1.4 extra wins per season just on the strength of going for it on 4th down when you're supposed to, even without knowing the other team's play call. Could this difference account for some of the extra two wins?

The problem is, it's not just the decision-makers who are risk-averse. If a coach won 4 games with this method and then lost 2 with it, the fans would be screaming for his head.

Elliot-Yeah, that was the basic premise of my article a while back on "gameday coaching." I had always heard that the Pats liked to run on 3rd and short more often than other teams too.

Since then, I've analyzed play-by-play data, and the Pats don't really stand out more than other teams in these regards. But this doesn't mean it isn't true. It could be the case, but I'm just not looking at it the right way.

The scientific research on this question is indisputable. I've yet to see one economist, mathematician or scientist publish a paper defending the status quo choice to always kick on fourth down. Yet it remains the status quo.

We know that coaches are aware of this research on at least some level. One thing they always say is "I wouldn't want to have to explain to the owner the next day why my call blew the game...."

Maybe, we are looking at the wrong person in the equation. Maybe the real question is why owners aren't hiring coaches that make the right decision. What would happen if an owner hired a coach and said "I will never second guess you if you make a solid calculated risk based on researched probabilities. But if you keep playing this weak-tight bullshit and passing up legitimate opportunities to win based on 'conservatism', you're out".

Frankly, I think owners are the perfect people to approach with this sort of plan. The owner is far enough removed from the game to make decisions in a calculated and mathematical way. He wants to win obviously. But the concept of losing one individual game and being banished from all future employment possibilities because of it is not a realistic fear.

Owners definitely get the concept of calculated risk. You do not get to be a billionaire without taking a few calculated risks along the way. In business, the people that don't take calculated risks and play it safe all the time ultimately become middle managers with a 401-k and dental. The ones that do take the risks earn the billions, even if they encounter a few sleepless nights along the way.

Frankly I think most owners would get it if they looked at the question in detail and I think that a prospective coach could actually win himself the job if he sat down the owner and explained that he understands calculated risk and exactly why it would be his policy to almost always go for it on 4th and goal from the 1 yard line. If the owner understood the policy and the logic beforehand, there would be little backlash from him on the occasions where the calculated risk comes up tails.

TMQ's explanation of conservatism is a little more cynical than that. He thinks it's just easier to avert blame if a coach plays it conservative than if he takes a risk that backfires.

His latest on the subject can be read here under "Tennessee" near the end.

This is a different topic but somewhat related to the "go for it or kick" question. Do you have any stats on how often fake punts work? Or even better specific fake punt plays?

My own opinion is that teams should try to run a direct snap to the upback fake punt far more often. I sort of believe that this might work even more than a traditional run on fourth and short would. Here is why:

In essense you are getting a power rushing play with short yardage with 8 offensive players and 6 defenders. How does this happen?

The punter and punt returner are both nowhere near the play. The punting team will also line up two gunners on each sideline. These men are typically covered by two defenders a piece. None of these people will be anywhere near a play.

In essense, the typical 11-11 lineup has been transformed into a 8-6 lineup. The element of surprise is just a bonus as the offensive team knows that their job is to plow ahead and block for the upback while the defenders have usually turned around and started moving backwards anticipating a kick.

Obviously, the offense is at somewhat of a disadvantage compared to a typical play because the special teams are not their usual offense.

On the other hand, the punt return team isn't the usual defense either. I'd think this would equal a push.

Its sort of taken as a given that fakes work only once in a blue moon because they really depend on the element of surprise. I wonder what would happen if a team tried this play an absurdly high percentage of the time they had a fourth and short. Say 1/3 of the time.

After you've run it two or three times in a year, you'd think the other team wouldn't be fooled. They'd watch it all week on tape and be ready. You couldn't run it anymore. Right? I'm not so sure. What would they do differently? Number one option is to keep the defenders at the line of scrimmage longer. This will obviously result in a much shorter average return on the 2/3 of the time that you do kick it. It still doesn't do anything to alleviate the basic advantage that you are now running an 8-6 running play instead of an 11-11 play. I think the play would still work often enough.

Of course, the defense could stop double covering the gunners and move those guys into the center of the field. However, the punting team could then react by running a regular punt every time the gunners are single covered. Punt coverage teams double cover for a reason. The gunners are very likely to force a fair catch if they don't. In essense, forcing this defensive adjustment would mean they are now punting a 100% of the time with a great coverage unit instead of 100% of the time with an average unit.

Single covering the gunners would open up other strategies as well. How about if the punter aimed the ball 30 yards down the field, relatively flat and towards the side instead of 50 yards down the field, high and in the middle? The gunner would now be able to get to it roughly the same time as the return man. The return man would have to pick it up (and risk a likely fumble) or move out of the way while the coverage unit let it roll an extra 25 yards.

The bottom line is that running an occasional direct snap to the upback in punt formation gives the punting team a lot of good options and the receiving team a lot of hard choices. It would seem to me as a perfect example of how to use game theory to put your opponent in a difficult position. Yet, its rarely used.

I'm curious to see stats on how often its worked when it has been used. I can't seem to find them anywhere. Do you have anything on them?

Dave-Yeah, I agree that was a terrible call by Fisher.

Jeff-That's a good question. The NFL official play descriptions say "(punt formation)" even when it's a fake, so it would be possible to go through and see what the success rates have been for various distances to go.

By going for the fake more often, you'd create a game theory effect of making the defense respect the possibility more. But then your punt coverages would be more successful. It would create a new equilibrium.

Or how about this? Every 4th down package is a hybrid package of regular offense and punt coverage guys. Any 4th down could go either way. There are special rules for punt formations, so this might not be possible.

"WP is the one and only correct utility function for any game, including football. Winning is all that matters, whether by 1 point or 100 points."

To first order, yes, but in reality winning a game isn't all that matters, because you're not playing a single game, you're playing a season.

This is especially true in long-season games like baseball, but there are always a few situations in the year in football where winning is not necessarily the best option. Some of those situations are trivial (week 17 games, etc.) but others are more complicated - should a coach try a quarterback sneak with a slightly-injured quarterback, for instance.

On second thought, many of the "runs" from punt formation were aborted plays such as muffed or bobbled snaps. It might be hard to differentiate between intentional fakes and aborted plays.

Anon-Great point. We see that every year in week 17 when playoff-bound starters are rested. My axiom should really say "assuming the only goal is to win the game..."

There must be some way of breaking out the stats for actual fakes vs. muffs.

Maybe this is like the 2 pt conversion where screwed up PATs actually get recorded as failed 2-pt attempts messing with the percentages for both.

If coaches can only see the stats on "runs from punt formation" they are definitely going to come to the conclusion not to fake it.

Nothing to add here, just wanted to say great article Brian. Also Jeff, I really like your idea of using the up-back more often. That really does sound like it might work, both as an effective 4th down strategy and to open up new options because of game theory.

To really complicate the defense of the short snap on 4th down, put a "wildcat type QB" as the short snap receiver/ blocker. If he faked receipt of the snap when it went to the punter, he would rarely have to block. He might take a shot or two.....

The receiving team's adjustment might be to blitz, but a short pass to the gunner on a slant or a swing pass as well as a punt would tend to offset the blitz the short guy approach.

A little OT Brian, but can you run the numbers on surprise onside kicks (Onsides in the first 3 quarters) and see what the success rate is and what the break even point is for onside kicks?

Anon- I've already done exactly that and have a couple articles on the topic already drafted. But not to be too coy, the answer is that the success rates are surprisingly high. I'll also have a 'value of a touchback' article coming up.

This reminds me of the decision I never understand in football. Perhaps you've written on it. It's the 4th quarter. You're down by 1. The other team is trying to run the clock out. They're on the 20 yard line (just into the redzone), you have no timeouts, and they just converted. Why do no coaches then just let the offense score? This may be ground well covered, but I can't imagine the win percentage is better when you must rely on blocking a field goal with 3 seconds left and returning it all the way, as opposed to relying on having a drive and 2 point conversion. The decision to just tackle them after the first down and essentially concede the game rankles.

As a follow on, I think it is because coaches feel that you should never willingly "lose" a play and it's just a bad precedent to set. However, you're coached to know your down and distance every time, and essentially concede good 8 yard passes on 3rd and 10+. Conceding a score late in the game is just a slightly larger take on that same concept.

I tend to think that if coaches started doing that, the other side would see what they were doing and run it just past the first down marker and then take a knee.

Then they wouldn't even have to take the risk of having the field goal blocked.

Of course this raises more interesting questions. What if it is 1st and goal?

At what point should the defending team let the offense run the ball in?

And the corollary...

At what point should the offense sit on the ball three times and kick on fourth?

Presumably, if its in the defense's interest to let them score. Its in the offense's interest to not accept that option.

In a way, it does make it more interesting. It would be weird to see a play in which the defenders didn't bother lining up at all or lined up 50 feet away from the ball and then the offense just took a knee. The first time you saw it, you'd find it amusing. If it became routine, you'd probably stop watching football out of frustration.

You could argue that this strategy is basically equivalent to intentionally fouling in basketball. Let them score so you get the ball back. Its not quite the same thing though since the other team needs to actually make the shots for it to work.

If this did become routine, I would guess that the competition committee would change the rules to avoid it.

How about if any team can basically buy the ball at any point in the game for 4 points?

Just go up to the referee and say "We'll give them 4 points in exchange for the ball". The other team has to accept.

Teams would never do this until they were desperate but once they were desperate, the viewers would still get to see an attempt at a comeback.

I perhaps stated it wrong, but I meant after the running back got the first. So, essentially, if it's 3rd and 1 at the 20, you completely sell out to stop the yard, and if the RB busts through, you really don't want to see your LB tackle him 5 yards downfield.

I've seen this in a few games each year, in which the last two minutes are simply one team running out the clock, and the other trying to keep them from a first down, when a first down is irrelevant. Were I a coach, I wouldn't make it obvious I was letting them score, I'd just sell out for a negative play, and if I didn't get it...they score, but it just looks like I was playing aggressive. The upside I get is I actually give myself a chance to win the game with most none the wiser.

Either way, the right answer can't be to tackle them 2 yards after they get the first and ensure you lose.

From the offensive side, it's trickier. Technically the RB should stop before the end zone, but it's even harder for most people to wrap their head around not scoring when they can, than giving up a score. I think RBs would only do that if they immediately went into victory formation (not 20 seconds left when they run out of that set of downs.)

"Why do no coaches then just let the offense score?"

The Packers did it in Super Bowl 32, with the game tied.

Good point Dave M. In that case it may not have been the right decision for a multitude of reasons...

If GB had stopped them, it would have been a field goal try with around a minute left, and GB then could have attempted to go down and score 3.

But, I like someone at least thinking it. The key is time and GB misestimated how much time they'd have.

But, you're right. It has happened. Just not well, or enough.

This is an absolutely fascinating analysis! But one problem I see at first blush is the failure to account for costs imposed by NFL social norms in the cost-benefit analysis. The culture is incredibly conservative because so much attention is focused on NFL coaching decisions and such vast sums of money are at stake. It's so conservative that coaches may be deterred by reputational costs from deviating from NFL conventional wisdom even where an unorthodox play has a higher expected value in yardage, points, etc..

Good point. You're right. They should totally sell out to stop the yard there.

I actually think there are a number of examples when teams have done this. I think they try to make it fairly subtle when they do though. They should do it more often.

You're right. Its a lot harder for people to get their heads around not scoring when they can. You also know that the RB is going to be thinking that 6 months from now, when he is negotiating a new contract, people aren't going to be as impressed with 14* touchdowns as they would be with 15.

Further to the question of letting the offense score and the comment that RB should stop at the one yard line, we have Brian Westbrook doing just that a few years ago. And it generated a lot of talk in the media.

If my memory serves me correctly Tom Landry let the 49ers score inside 2 minutes of a playoff game many years ago. Either I'm making all this up but as I remember it, the Cowboys got the ball and ran the field for the win.

Landry, over the course of his carrier tried a number of aggressive ideas. My favourite was the split line. He wold have the line offensive line take their usual positions but there would be a 10 yard gap between the guard and center. IT was a crazy formation but it made for some interesting football. If you've never seen it try get to some old film.

Just found this web site. Interesting application of decision theory.

Two questions:

1. How is a picnic worth 4 units of enjoyment while bowling is worth 1 unit? Is a picnic really 4x more enjoyable than bowling? How would anyone non-subjectively determine that?

2. The decision theoretic approach might be effective for a simple yes/no choice like determining whether or not to go for it on 4th down. However: A) could the numbers be skewed by the fact that it is uncommon to go for it on fourth down? If all coaches adopt decision theoretic strategies, might the odds of success change based other teams's expectations that decision theoretic calculus will be used. B) The situation would be even worse for decision theoretic play calling, where again, decision theoretically informed coaches would have a better idea of what's coming from the opposing decision theory minded coach. A lot of football play calling depends on mismatches and the element of surprise. Would this be lost if every coach was using that same mathematical model?

Considering the approach of a team 'letting' the other team score...

You are forgetting that he doesnt HAVE to take a knee. Once its happened once there would be enough coverage of it within each teamt that an RB would run up to the 2 yard line and run around in circles... winding up the clock and not score.

There was a game a couple of seasons ago... Eagles vs Cowboys & Westbrook slid on the 1 yard line instead of running it in, very late in the 4th. Scoring there would have made it a 2-possession game, but given the Cowboys the ball back, instead he slid, took one for the team & they just took three knees & the game was over.

Intelligent, selfless play... should be more of it in the NFL.

Steven.

I would say that you need to dive even deeper into your analysis to get the true gist of what all this really means? Tactical football requires an analysis that is independent of statistical analysis. It is more about the intent of the coach through his use tactical schematic reasoning. His decisions will be based on his confidence in the execution of the offense and defense. Situational football can be analyzed left and right but there are some stats that are more important than others. Your 4th down analysis has to be originated from the previous 3rd down. Coaches make decisions on the 2nd before the 1st down is over and the same can be applied to 2nd and 3rd down situations. His confidence in his decision will be predicated on his experience in schematic reasoning not statistical. There is an obvious parallel but it is a false positive. It is all about the scheme which will influence all of his tactical decisions.

William, you just wrote a whole lot of words and said essentially nothing. You have no grasp of what you are talking about and are just trying to sound smart.

I think what William Brown is saying is that the coach uses a theoretical/intuitional understanding that encompasses more variables than the statistical analysis possibly can at this point (such as how his players are performing at different positions). While this is true, and coaching is still somewhat an art, statistical analysis can be added to this and inform it. You can decide to go for it on fourth down based on stats (assuming your offense doesn't completely suck, though at the NFL level things are more reasonably level) and then decide based on other factors where to run it. The problem though is that much of the results of intuitional analysis is illusory. This can be helped by statistically analyzing your own team.