## Simulating the Saints-Falcons Endgame

I was asked yesterday about the end of regulation of the Saints-Falcons game. With about a minute and a half remaining, NO was down by 4 but had a 1st & goal at the 1. With 2 timeouts left, should ATL have allowed the touchdown intentionally?

I previously examined intentional touchdown scenarios, but only considered situations when the offense was within 3 points. In this case NO needed a TD, which--needless to say--makes a big difference. Yet, because NO was on the 1, perhaps the go-ahead score was so likely that ATL would be better off down 3 with the ball than up 4 backed-up against their goal line.

This is a really, really hard analysis. There's a lot of what-ifs: What if NO scores on 1st down anyway? What if they don't score on 1st but on 2nd down? On 3rd down? On 4th down? Or what if they throw the ball? What if they stop the clock somehow, or commit a penalty? How likely is a turnover on each successive down? You can see that the situation quickly becomes an almost intractable problem without excessive assumptions.

That's where the WOPR comes in. The WOPR is the new game simulation model created this past off-season, designed and calibrated specifically for in-game analytics. It simulates a game from any starting point, play by play, yard by yard, and second by second. Play outcomes are randomly drawn from empirical distributions of actual plays that occurred in similar circumstances.

If you're not familiar with how simulation models work, you're probably wondering So what? Dude, I can put my Madden on auto-play and do the same thing. Who cares who wins a dumb make-believe game?

Fair point, but here's a cool story. During the Manhattan project, scientists needed to be able to solve really nasty high-dimension integrals. Direct analytic solutions were impossible, so the scientists developed a method of randomly sampling values within the problem. By repeating this enough times, they could get increasingly solid estimates for the answer they needed. Because of the randomness at the core of the technique, they called it Monte Carlo integration.

Simulation models generally work the same way. It's run not once but many, many times. The proportion of times in which a chosen event occurs in the simulation can be an estimate of the probability of that event. For example, the proportion of times the sim says that ATL wins starting at their own 20 down 3 points with 2 timeouts and 1:24 left on the clock is an estimate of their win probability for intentionally allowing the TD. And the proportion of times ATL wins starting up 4 points with their backs against the goal line is their win probability estimate for playing straight up defense.

In this case, virtual ATL won 25.3% of the time by playing defense and 16.0% of the time by intentionally allowing the TD. That's a pretty big spread as far as these things go, so the numbers support ATL's approach. (I'm not even sure if it was a decision. The announcers on Fox raised the question, so I presume it crossed the minds of the ATL staff.)

It might be tempting to plug the situation into the (empirical) WP model, but situations like this pose two problems that combine to make things tough. They are simultaneously very rare and have high leverage on the game outcome. One advantage of a sim model is that you can create as large a sample as you need for almost any situation you can imagine. Happily, the WP model yields the same answer with approximately the same results (24.9% and 16.9%).

Another advantage of the WOPR is that it can provide insights into the reasons for the results. I ran the simulation one "game" at a time and looked at a sample of how the games unfolded. In cases where ATL won by playing straight defense, something unfortunate often happened to NO--they got penalized, turned the ball over, or took a loss (which actually occurred). It's these kinds of complications that are often the first things to be assumed away for the purpose of making the problem directly solvable.

The insights are often the real value of any good model. The answers to the questions are great, but gaining a fundamental understanding about how the sport really works is even better.

### 4 Responses to “Simulating the Saints-Falcons Endgame”

1. Unknown says:

With the clocking running, what was the break-even? I wanted Atlanta to let them score on the first down because converting a yard is fairly easy.

2. Brian Burke says:

Andrew-That's a good question, but it points to one of the difficulties with a sim model, it takes a lot of time, especially running multiple cases with varying initial conditions.

3. Anonymous says:

lol. wow. so this is the manhattan project.

um .. i don't think so.

4. Unknown says:

No...not the Manhatten project. Down four in this scenario, the best play on 1st is to try make sure you don't give up a soft TD on the ground; encourage Off to pass or run round the edge or at least force a hand-off rather than a QB sneak. Do something might force an error; pressure might even induce a FS or DoG penalty. Sometimes the mentality pays dividends over the literal maths.

Sometimes such scenarios may occur because of a more fundamental error earlier in the game