Simulating the Saints-Falcons Endgame

I was asked yesterday about the end of regulation of the Saints-Falcons game. With about a minute and a half remaining, NO was down by 4 but had a 1st & goal at the 1. With 2 timeouts left, should ATL have allowed the touchdown intentionally?

I previously examined intentional touchdown scenarios, but only considered situations when the offense was within 3 points. In this case NO needed a TD, which--needless to say--makes a big difference. Yet, because NO was on the 1, perhaps the go-ahead score was so likely that ATL would be better off down 3 with the ball than up 4 backed-up against their goal line.

This is a really, really hard analysis. There's a lot of what-ifs: What if NO scores on 1st down anyway? What if they don't score on 1st but on 2nd down? On 3rd down? On 4th down? Or what if they throw the ball? What if they stop the clock somehow, or commit a penalty? How likely is a turnover on each successive down? You can see that the situation quickly becomes an almost intractable problem without excessive assumptions.

That's where the WOPR comes in. The WOPR is the new game simulation model created this past off-season, designed and calibrated specifically for in-game analytics. It simulates a game from any starting point, play by play, yard by yard, and second by second. Play outcomes are randomly drawn from empirical distributions of actual plays that occurred in similar circumstances.

If you're not familiar with how simulation models work, you're probably wondering So what? Dude, I can put my Madden on auto-play and do the same thing. Who cares who wins a dumb make-believe game? 

Analyzing Replay Challenges

The new WP model allows some nifty new applications. One of the more notable improvements is the consideration of timeouts. That, together with enhanced accuracy and precision allow us to analyze replay challenge decisions. Here at AFA, we've tinkered with replay analysis before, and we've estimated the implicit value of a timeout based on how and when coaches challenge plays. But without a way to directly measure the value of a timeout the analysis was only an exercise.

Most challenges are now replay assistant challenges--the automatic reviews for all scores and turnovers, plus particular plays inside two minutes of each half. Still, there are plenty of opportunities for coaches to challenge a call each week.

The cost of a challenge is two-fold. First, the coach (probably) loses one of his two challenges for the game. (He can recover one if he wins both challenges in a game.) Second, an unsuccessful challenge results in a charged timeout. The value of the first cost would be very hard to estimate, but thankfully the event that a coach runs out of challenges AND needs to use a third is exceptionally rare. I can't find even a single example since the automatic replay rules went into effect.

So I'm going to set that consideration aside for now. In the future, I may try to put a value on it, particularly if a coach had already used one challenge. But even then it would be very small and would diminish to zero as the game progresses toward its final 2 minutes. In any case, all the coaches challenges from this week were first challenges, and none represented the final team timeout, so we're in safe waters for now.

Every replay situation is unique. We can't quantify the probability that a particular play will be overturned statistically, but we can determine the breakeven probability of success for a challenge to be worthwhile for any situation. If a coach believes the chance of overturning the call is above the breakeven level, he should challenge. Below the breakeven level, he should hold onto his red flag.

When Coaches Use Timeouts

As I continue to work on the next generation WP model, I'm looking hard at how timeouts are used. Here are 2 charts that capture about as much information as can be squeezed into a graphic.

The charts need some explanation. They plot how many timeouts a team has left during the second half based on time and score. Each facet represents a score difference. For example the top left plot is for when the team with the ball is down by 21 points.  Each facet's horizontal axis represents game minutes remaining, from 30 to 0. The vertical axis is the average number of timeouts left. So as the half expires, teams obviously have fewer timeouts remaining.

The first chart shows the defense's number of timeouts left throughout the second half based on the offense's current lead. I realize that's a little confusing, but I always think of game state from the perspective of the offense. For example, the green facet titled "-7" is for a defense that's leading by 7. You can notice that defenses ahead naturally use fewer timeouts than those that trail, as indicated by comparison to the "7" facet in blue. (Click to enlarge.)

The Value of a Timeout - Part 2

In the first part of this article, I made a rough first approximation of the value of a timeout. Using a selected subsample of 2nd half situations, it appeared that a timeout's value was on the order of magnitude of .05 Win Probability (WP). In other words, if a team with 3 timeouts had a .70 WP, another identical team in the same situation but with only 2 timeouts would have about a .65 WP.

In this part, I'll apply a more rigorous analysis and get a better approximation. We'll also be able to repeat the methodology and build a generalized model of timeout values for any combination of score, time, and field position.

Methodology

For my purposes here, I used a logit regression. (Do not try to build a general WP model using logit regression. It won't work. The sport is too complex to capture the interactions properly.) Logit regression is suitable in this exercise because we're only going to look at regions of the game with fairly linear WP curves. I'm also only interested in the coefficient of the timeout variables, the relative values of timeout states, and not the full prediction of the model.

I specified the model with winning {0,1} as the outcome variable, and with yard line, score difference, time remaining, and timeouts for the offense and defense as predictors. The sample was restricted to 1st downs in the 3rd quarter near midfield, with the offense ahead by 0 to 7 points.

Results

The Value of a Timeout - A First Approximation

During the NFC Championship Game the other day, we saw a familiar situation. Down by 4 with 14 minutes left in the game, the Seahawks were confronted with a decision. It was 4th and 7 on the SF 37. Should they go for it, punt, or even try a long FG to maybe make it a 1-point game? Pete Carroll ended up making what was the right decision according to the numbers, but not before calling a timeout to think it over.

As I noted in my game commentary, if you need to call a timeout to think over your options, the situation is probably not far from the point of indifference where the options are nearly equal in value. And timeouts have significant value, particularly in situations like this example--late in the game and trailing by less than a TD--because you'll very likely need to stop the clock in the end-game, either to get the ball back or during a final offensive drive. Would Carroll have been better off making a quick but sub-optimum choice, rather than make the optimum choice but by burning a timeout along the way?

Here's another common situation. A team trails by one score in the third quarter. It's 3rd and 1 near midfield and the play clock is near zero. Instead of taking the delay of game penalty and facing a 3rd and 6, the head coach or QB calls a timeout. Was that the best choice, or would the team be better off facing 3rd and 6 but keeping all of its timeouts?

Both questions hinge on the value of a timeout, which has been something of a white whale of mine for a while. Knowing the value of a timeout would help coaches make better game management decisions, including clock management and replay challenges.

In this article, I'll estimate the value of a timeout by looking at how often teams win based on how many timeouts they have remaining. It's an exceptionally complex problem, so I'll simplify things by looking at a cross section of game situations--3rd quarter, one-score lead, first down at near midfield. First, I'll walk through a relatively crude but common-sense analysis, then I'll report the results of a more sophisticated method and see how both approaches compare.

Payton Was Right to Decline

At least according to Expected Points, he was.

Here's the situation: At the beginning of the 3rd quarter against CAR, NO had a 1st and 10 at their own 16-yard line. They threw for a 7-yard gain, setting up a 2nd and 3 from their 23. But CAR was flagged for defensive holding, which would have given NO 5 yards and an automatic first down at their 21. NO head coach Sean Payton declined the penalty to the bafflement of many including the tv announcers.

The game did not hinge on this decision by any stretch. But it's worth taking a look at. The EP model is probably the right tool for the job in this case because it gives a much finer level of precision to down/distance/yd-line situations than the WP model or other approaches.

Using the hand-dandy WP calculator tool (which as a bonus is also an EP calculator), here are the relevant numbers:

Seahawks Stumble, Should Have Allowed TD

In one of the most anticipated games of the week, the San Francisco 49ers took over down 17-16 to the Seattle Seahawks with 6:20 remaining. After a huge Frank Gore 51-yard run, the Niners lined up for a 1st-and-Goal from the 7-yard line with 2:39 remaining. Seattle had no timeouts remaining. Should the Seahawks have tried to intentionally allow the Niners to score a touchdown? Let's look at Brian's graph for this situation in his intentional TD study:

Is the Revolution Over? Did We Win?

"The Revolution Was Televised. The fourth down revolution is over. Going for it won."

-Mike Tanier, Sports on Earth

Is Mike right? Did going for it really win? Mike makes a the case, and cites several promising examples of unconventional 4th down decisions from one Sunday afternoon earlier this season:

"-The Lions going for it on 4th-and-goal from the two-yard line, early in their win over the Cowboys.
-The Dolphins going for it on 4th-and-1 from the Patriots' 38-yard line, in the second quarter.
-The Patriots going for it on 4th-and-4 from the Dolphins' 34-yard line, while leading by three points in the fourth quarter.
-The Bengals going for it on 4th-and-inches from the 1-yard line, while leading 14-0 against the Jets.
-The Broncos scoring a 4th-and-goal touchdown to tie the game at 21 against the Redskins, in the third quarter.
-The Packers converting a 4th-and-3 from their own 42-yard line, setting up a touchdown to increase their lead to 31-17."

I think Mike is right to point out some very interesting cases where coaches are making some notable decisions, but the revolution is far from complete. I would suggest that an avalanche is the better analogy than revolution. One day there may be an avalanche of aggressive 4th down decisions, but right now we're only seeing a few rocks trickle down the mountainside. It's not that there haven't been bold examples of enlightenment. It's just that there are so many opportunities that coaches have spurned.

When the Defense Should Decline a Penalty After a Loss Part 2 (2nd Downs)

I recently looked at when it made sense for the defense to decline a 10-yard holding penalty following a 1st down play for no gain or a loss. It turned out that defenses should generally prefer to decline after a loss of 3 or more yards.

First downs are easier to analyze because they almost always begin with 10 yards to go. Unfortunately, 2nd downs aren't so cooperative. It's amazing how thin he data gets sliced up. Most downs aren't losses, even fewer have holding penalties, and rarely are they declined. Still, there are enough cases for a solid analysis using 1st-down conversion probability as the bottom line.

Put simply, a defense would prefer to decline a penalty on a 2nd down play whenever the resulting 3rd down situation leads to a conversion less often than the 2nd down plus the 10 yards.

The chart below plots conversion probability for 2nd and 3rd down situations. The red line illustrates the conversion probability of 3rd down and X to go situations. For example, 3rd down and 7 situations are converted about 40% of the time.

The green line illustrates 2nd down situations, but slightly differently. It plots conversion probabilities for 2nd down and X plus 10 yards. For example, 2nd and 13 (i.e. 3 + 10 yds) situations are converted 45% of the the time. The black line is the smoothed line fitted to the 3rd down conversion rates. I plotted things this way because it's the actual comparison we're interested in, given a gain of zero yards.

On the Effect of Coaching

There are exceptions, like that guy on the left, but my hunch is that NFL coaches are mostly interchangeable.

I think at the NFL level, all coaches employ the same best practices. There is no secret sauce that one coach has over another in terms of instruction, motivation, strategy, etc. This is because of the highly mobile, fluid market for coaches and the large size of their staffs. There are very strong constraints on deviation from league norms in any dimension.

Also, from statistical analysis, we can measure the variance in team performance attributable to randomness (sample error due to a short 16-game season) and player impacts (the addition or subtraction of a player's impact on team production, player interaction effects). There is very little variance left that can be attributed to other causes, including coaching. In other words NFL outcomes are overwhelmingly driven by player talent and luck, and there's not much room left for coaching to make a big impact.

We can observe this intuitively, as the very same coaches can have wildly different records from year to year. How much effect can they really have?

When Should the Defense Decline a Penalty After a Loss? Part 1

Let's say there's a sack or other tackle that results in a several-yard loss. And to compound the offense's woes a flag for holding is thrown, potentially setting up a 1st and 20 situation. Should the defense accept or decline the penalty and force a 2nd and X? We can evaluate this question in a few ways. We'll use a simple method and a more complex method to find out when a defense should normally decline a penalty on first down.

Before you read on, what do you think the break-even yardage is? What do you think most coaches think it is?

Examining the Value of Coaches' Challenges

Kevin Meers is the Co-President of the Harvard Sports Analysis Collective. He is a senior majoring in economics with a statistics minor, and has spent the past two years or so as an analytics intern in the NFL. He is currently writing his thesis on game theory in the NFL, and probably puts too much thought into how the perfect fantasy football league would be structured.

The coach’s challenge is an important yet poorly understood part of the NFL. We know challenges are an asset, but past that, we do not have a good understanding of what makes a good challenge or if coaches are actually skilled at challenging plays. This post takes a step towards better understanding those questions by examining the value of the possible game states that stem from challenged plays.

To value challenges, we must understand how challenges change the game’s current state. When a play is challenged, the current game state must transition into one of two new game states: one where the challenged play is reversed, the other where it is upheld. These potential game states are the key to valuing challenges.

Let’s look at a concrete example from last season. With two minutes and two seconds left in the fourth quarter in their week ten matchup, Atlanta had first and goal on New Orleans’ ten-yard line. Matt Ryan completed a pass to Harry Douglas, who was ruled down at the Saints’ one-yard line… only Douglas appeared to fumble as he went to the ground, with the Saints recovering the ball for a potential touchback. When New Orleans challenged the ruling on the field, the game could have transitioned into two possible game states: Atlanta’s ball with second and goal on the one, or New Orleans’ ball with first and ten on their own 20 yard line. If the Saints lost the challenge, they would have a Win Probability (WP) of 0.28, but if they won, their WP would jump to 0.88. This potential WP added, which I refer to as “leverage,” is key to valuing challenges. Mathematically, I define leverage as:

Should You Bench Your Fumbling Running Back?

Sam Waters is the Managing Editor of the Harvard Sports Analysis Collective. He is a senior economics major with a minor in psychology. Sam has spent the past eight months as an analytics intern for an NFL team. When he is not busy sounding cryptic, he is daydreaming about how awesome geospatial NFL data would be. He used to be a Jets fan, but everyone has their limits.

When the Pittsburgh Steelers traveled to Cleveland in week 12 of last season, Rashard Mendenhall was the Steelers’ starting running back. Well, he was at first. Mendenhall fumbled on his second carry of the game, and Head Coach Mike Tomlin benched him immediately. On came backups Isaac Redman, Jonathan Dwyer, and Chris Rainey, who all fumbled and joined Mendenhall on the sidelines in quick succession. Out of untainted running backs to sub in, Tomlin looped back around to Mendenhall, who put the ball on the ground again. Mendenhall, of course, went right back to the bench, ceding his snaps to Dwyer and Rainey for the rest of the game. This was one of the more prolific fumble-benching sprees in NFL history, but we see tamer versions of this scenario all the time. Just look back to last season. David Wilson fumbled and Tom Coughlin actually made him cry. Ryan Mathews fumbled away his job to Jackie Battle. Tears and Jackie Battle - does any mistake deserve these consequences?

Rex Ryan Runs Out of Challenges

Early in the 4th quarter of Sunday's BUF-NYJ game, the Jets led by 8. BUF QB EJ Manuel apparently lost a fumble, but the officials ruled him down. Had NYJ recovered, they would have had the lead and a 1st and 10 in BUF territory, only about 5 yards from realistic FG (attempt) range, with about 13 minutes left to play. The only problem was that NYJ head coach Rex Ryan had no more challenges, having burned them both just minutes prior to the fumble that wasn't.

Let's examine the leverage of each challenge using the Win Probability (WP) model. 

When to Intentionally Allow a TD When Tied

Super Bowl 32 was a memorable one. A tight game featuring John Elway and Brett Favre would have made a memorable regular season game, but as a Super Bowl it was spectacular. To me the most interesting thing about the game was how the winning score happened. It was allowed intentionally.

The game was tied at 24. The Broncos began a drive with 3:27 left to play. After a big Elway pass and several Terrell Davis runs, Denver put Green Bay in the Field Goal Choke Hold. Eventually, Denver fought its way to a 1st and goal from the Green Bay 8. A holding call on Shannon Sharpe moved Denver back to 1st and goal from the 18. Another Davis run set up 2nd and goal from the 1 with just 1:47 to play. Rather than allow Denver to run down the clock any further, head coach Mike Holmgren elected to allow the TD on the next play to give his offense a better chance to respond with a TD of their own.

In the wake of my previous five-part analysis of intentionally allowing a TD, I learned what the Internet jargon tl;dr stands for. I promise to make this one shorter. Previously, I looked at situations in which an offense that's trailing by 1 or 2 points could run out the clock before kicking a field goal to win. In many cases, depending on the time, score, field position, and number of timeouts remaining, it makes sense for the defense to allow a TD rather than try to force a stop and a FG attempt.

This time I'll examine similar situations where the score is tied. The considerations are a little different than when the defense has a 1 or 2 point lead. A tie score means that the defense can't be relatively assured of a win in the event of a miss. And given a successful FG to break a tie, a FG in response only re-ties the game.

New Overtime 4th Down Decisions When Down 3 Points

Your opponent kicked a FG on the first possession of overtime, and now your team needs a TD to win or a FG to continue the game. Your offense has driven down to the opponent's 10-yard line, but the drive has stalled. It's 4th down and 3. Should you go for the risky conversion and ultimately a TD for the win, or should you attempt a FG knowing you'd be at a disadvantage giving the ball to the opponent in sudden death?

The new NFL OT rules are unique in a lot of ways, and by unique I mean convoluted and contrived. There are basically three possible game states:

1. The first drive in which no score leads to Sudden Death, a TD wins, or a FG spawns the second state...
2. A possible second possession in which the offense is down by 3 points. It must score a TD to win or a FG to continue into SD.
3. Lastly, traditional SD itself.

The three game states successively easier to model. The first possession must consider all the possibilities of the following two states. The second state must only consider itself and the possibility of SD. The second possession is also slightly easier to model because there is no punt option. An offense trailing by 3 points simply must score or lose.

New Feature: Time Calculator

Someone tell Norv Turner there are no timeouts in press conferences.

I created a new tool to estimate the time at which a trailing defense (or soon-to-be-trailing defense) can get the ball back if they force a stop. The results are based on the time at the first down snap of a series and the number of timeouts remaining for the defense. You can adjust the expected duration of each play and the time consumed between the previous whistle and the next snap when the game clock is not stopped. The defaults are 6 and 39 seconds respectively. The calculator assumes there will be no stoppages due to reasons other than timeouts and the two-minute warning, such as incomplete passes, runs out of bounds, or penalties.

One additional feature is that you can check a box called "Save Timeout." This will indicate that the team on defense would prefer to allow the clock to wind down to the two minute warning rather than stop the clock with a timeout. For example, if the defense has one timeout left and the second down play ended at 2:10, the defense can elect to save the timeout for its offense in exchange for running down the 10 seconds to 2:00. This is, in effect, a trade-off between the 10 seconds of game clock and having a timeout available for an offensive drive.

It's very difficult to quantify the value of the timeout on offense. It's intuitively very valuable because an offense can use the middle of the field, which otherwise allows the defense to guard the sidelines.

Try this: Enter 2:24 remaining with 3 timeouts. Leave the 6 sec and 39 sec defaults for play and inter-play durations. Click calculate with the Save Timeout option unchecked and checked (with the 12-second default cutoff value). With 'Save Timeout' checked, you get the ball back with 1:54 and retain a  timeout for your offense. Without the option checked, you get the ball back with 2:00 on the clock and no timeouts, with the 2-minute warning essentially going to waste.

This option usually only makes a difference when the defense begins the series with all three timeouts remaining. It also may be smart depending on when a team can expect the change of possession to occur. The defense does not want change of possession to occur on a play that spans the two minute warning because that combines two potential clock stoppages into a single stoppage.

The workings of the NFL game clock is far more complex than it might seem. That's why I forced myself to build the calculator and think through all the considerations. The algorithm behind the calculator is basically a by-product of the one I used to create the chart below, which underpinned my analysis of when a defense should prefer to intentionally allow a TD.

Fourth Downs in the New Overtime: First Possession

This may have been the most difficult, challenging analysis I've done. No joke. The new OT format is more complex than it seems. There are three distinct 'game states' in which a team can find itself:

1. The initial drive of the first possession (A TD wins, a turnover or punt triggers Sudden Death (SD), and a FG triggers State 2.)
2. The team down by 3 now has one possession to match the FG (triggering SD) or score a TD to win.
3. Sudden Death

The possibilities are illustrated in the event tree below, along with some back-of-the-napkin transition probabilities I made back when the new rules were first proposed. (State 1 is "1st Poss". State 2 is the branch under "2nd Poss" that follows a FG in the 1st Poss. Sudden death is self-explanatory and occurs after a no-score in the 1st Poss or after a FG is matched in the 2nd Poss.)

One Play Remaining before the Half on the Goal Line

IND QB Andrew Luck spiked the ball with 1 sec remaining in the 2nd quarter, bringing up a 3rd and goal from the 1-yd line. Without a moment of hesitation, acting head coach Bruce Arians ran in the FG unit for the chip-shot. The FG was good and IND took at 13-6 lead over BUF into the locker room. Was this the smart call?

Let's set aside the score and look at the general case. It's a special situation because there is no subsequent kickoff. Instead of being worth 2.7 Expected Points (EP), a FG is worth a full 3 EP. And a TD would be worth a full 7 EP instead of 6.7. The offense would take the full value of the score.

The expected value of each choice is straightforward. It's just the probability of success * the value of the score. In the case of the FG it would be:

How Much Did Jim Schwartz's Attempted Challenge Cost the Lions?

Had Schwartz not thrown the challenge flag on Forsett's run, the play would have been reviewed and certainly overturned. That would mean a 3rd and 2 for HOU on their own 27 with 6:40 or so in the 3rd quarter. Down by 10 points, that means at 0.18 Win Probability (WP) for HOU.

But because Schwartz threw the challenge flag on a play that would have been otherwise reviewed automatically, he received an unsportsmanlike penalty. The result was that the play was not reviewed by rule, and Forsett's TD stood. That gave DET a touchback up by 3, giving HOU a 0.35 WP.

That's a cost of 0.17 WP. It essentially doubled HOU's chances of winning at that point.