Chip's Challenging Decisions

The Eagles lost to the 49ers on Sunday 26-21 in what essentially came down to two plays by the Eagles on 3rd and 4th-and-Goal from just outside the 1-yard line. There is no denying it was a close game - even though the Eagles offense was unable to move the ball all day. The Eagles made a few interesting decisions - or lack of decisions - in the second half including not challenging a 3rd-down conversion reception and accepting a penalty on 3rd-and-3 after an incompletion. Let's examine each and see how they could have affected the Eagles' win probability, keeping in mind that in close games, every percentage point counts.

Throw The Red Flag

With 2:09 left in the 3rd quarter, winning 23-21, San Francisco faced a 3rd-and-9 from their own 27. Colin Kaepernick dropped back and hooked up with Anquan Boldin for a 12-yard completion. It appeared Boldin bobbled the ball and potentially trapped it on the completion. In today's game where most of the biggest plays are automatically reviewed (scoring plays and turnovers), a long 3rd-down conversion in a close game is one of the higher leverage challenge situations.

Simulating the Saints-Falcons Endgame

I was asked yesterday about the end of regulation of the Saints-Falcons game. With about a minute and a half remaining, NO was down by 4 but had a 1st & goal at the 1. With 2 timeouts left, should ATL have allowed the touchdown intentionally?

I previously examined intentional touchdown scenarios, but only considered situations when the offense was within 3 points. In this case NO needed a TD, which--needless to say--makes a big difference. Yet, because NO was on the 1, perhaps the go-ahead score was so likely that ATL would be better off down 3 with the ball than up 4 backed-up against their goal line.

This is a really, really hard analysis. There's a lot of what-ifs: What if NO scores on 1st down anyway? What if they don't score on 1st but on 2nd down? On 3rd down? On 4th down? Or what if they throw the ball? What if they stop the clock somehow, or commit a penalty? How likely is a turnover on each successive down? You can see that the situation quickly becomes an almost intractable problem without excessive assumptions.

That's where the WOPR comes in. The WOPR is the new game simulation model created this past off-season, designed and calibrated specifically for in-game analytics. It simulates a game from any starting point, play by play, yard by yard, and second by second. Play outcomes are randomly drawn from empirical distributions of actual plays that occurred in similar circumstances.

If you're not familiar with how simulation models work, you're probably wondering So what? Dude, I can put my Madden on auto-play and do the same thing. Who cares who wins a dumb make-believe game? 

Analyzing Replay Challenges

The new WP model allows some nifty new applications. One of the more notable improvements is the consideration of timeouts. That, together with enhanced accuracy and precision allow us to analyze replay challenge decisions. Here at AFA, we've tinkered with replay analysis before, and we've estimated the implicit value of a timeout based on how and when coaches challenge plays. But without a way to directly measure the value of a timeout the analysis was only an exercise.

Most challenges are now replay assistant challenges--the automatic reviews for all scores and turnovers, plus particular plays inside two minutes of each half. Still, there are plenty of opportunities for coaches to challenge a call each week.

The cost of a challenge is two-fold. First, the coach (probably) loses one of his two challenges for the game. (He can recover one if he wins both challenges in a game.) Second, an unsuccessful challenge results in a charged timeout. The value of the first cost would be very hard to estimate, but thankfully the event that a coach runs out of challenges AND needs to use a third is exceptionally rare. I can't find even a single example since the automatic replay rules went into effect.

So I'm going to set that consideration aside for now. In the future, I may try to put a value on it, particularly if a coach had already used one challenge. But even then it would be very small and would diminish to zero as the game progresses toward its final 2 minutes. In any case, all the coaches challenges from this week were first challenges, and none represented the final team timeout, so we're in safe waters for now.

Every replay situation is unique. We can't quantify the probability that a particular play will be overturned statistically, but we can determine the breakeven probability of success for a challenge to be worthwhile for any situation. If a coach believes the chance of overturning the call is above the breakeven level, he should challenge. Below the breakeven level, he should hold onto his red flag.

Implications of a 33-Yard XP

The NFL is experimenting with a longer XP this preseason. XPs have become so automatic (close to 99.5%) that there no longer much rationale for including them in the game. The Competition Committee's experiment is to move the line of scrimmage of each XP to the 15-yard line, making the distance of each kick 33-yards.

Over the past five seasons, attempts from that distance are successful 91.5% of the time. That should put a bit of excitement and drama into XPs, especially late in close games, which is what the NFL wants. But it might also have another effect on the game.

Currently, two-point conversions are successful at just about half that rate, somewhere north of 45%. The actual rate is somewhat nebulous, because of how fakes and aborted kick attempts into two-point attempts are counted.

It's likely the NFL chose the 15-yd line for a reason. The success rates for kicks from that distance are approximately twice the success rate for a 2-point attempt, making the entire extra point process "risk-neutral." In other words, going for two gives teams have half the chance at twice the points.

When Coaches Use Timeouts

As I continue to work on the next generation WP model, I'm looking hard at how timeouts are used. Here are 2 charts that capture about as much information as can be squeezed into a graphic.

The charts need some explanation. They plot how many timeouts a team has left during the second half based on time and score. Each facet represents a score difference. For example the top left plot is for when the team with the ball is down by 21 points.  Each facet's horizontal axis represents game minutes remaining, from 30 to 0. The vertical axis is the average number of timeouts left. So as the half expires, teams obviously have fewer timeouts remaining.

The first chart shows the defense's number of timeouts left throughout the second half based on the offense's current lead. I realize that's a little confusing, but I always think of game state from the perspective of the offense. For example, the green facet titled "-7" is for a defense that's leading by 7. You can notice that defenses ahead naturally use fewer timeouts than those that trail, as indicated by comparison to the "7" facet in blue. (Click to enlarge.)

Thomas Bayes Would Approve of Seattle's Defensive Tactics

The following is a guest article by Gary Montry, a professional applied mathematician. Editor's note: Gary uses net yardage as the measure of utility, and we might prefer something like EP or WP, I think the general point of the article stands, and its strength is in the construction and solution to the problem. It's also a great refresher on conditional probabilities and Bayes' theorem.   

Last week a WSJ article about the Seahawks' defensive backs claimed that they "obstruct and foul opposing receivers on practically every play."  I took a deeper look in to the numbers and found that as long as referees are reluctant to throw flags on the defense in pass coverage (as claimed in the article), holding the receiver is a very efficient defensive strategy despite the risk of being penalized.

The following is an analysis using the concepts of expected utility, expected cost, and bayesian statistics.

The reason defensive holding is an optimal strategy comes down to one word. Economics. The referee's reluctance to call penalties on the defensive secondary is analogous to a market inefficiency. The variance in talent on NFL rosters, coaching staffs, and front offices between the best and worst teams in the league is probably very small. Successful teams win within a small margin. Seattle has found a way to exploit a relaxation in marginal constraints within the way the game is called that their competitors have not, and turned it into a competitive advantage.

If you think about committing a penalty in the same way as committing a crime, the expected utility is essentially the same. The expected utility (EU) for defensive holding is (opponent loss of down due to incomplete pass - probability of being penalized x cost of penalty). In other words, EU is the benefit of an incomplete pass minus the cost of the penalty times the probability of getting caught.

NFL Overtime Modeled as a Markov Chain

by Ben Zauzmer. Ben is a junior majoring in Applied Math at Harvard University and is a member of the Harvard Sports Analysis Collective. This article was originally published at harvardsportsanalysis.org.

In 2012, the NFL created new overtime rules designed to make the game fairer. The league switched from a sudden death setup to an arrangement that allows both teams to have a chance at scoring, unless the first team to receive scores a touchdown. Even with this change, it would seem that a coach should still always elect to receive if he wins the coin toss at the start of overtime, since an opening touchdown drive wins the game.

However, earlier this year, for the first time under the new rules, a coach made exactly the opposite decision. Bill Belichick, the three-time Super Bowl-winning coach of the New England Patriots, made the gutsy call to kick at the start of overtime. Many considered the main factor behind this decision to be the heavy winds at Gillette Stadium (if a team defers the choice of kicking or receiving, it may choose which direction to face). However, kicking first may also give a team better field position on offense and may actually benefit teams with strong defenses.

To calculate which strategy coaches should prefer, we will model NFL overtime as a Markov Chain. We will define our states as the set of possible point differentials, from the perspective of the team that receives the opening kickoff, in overtime: -6, -3, -2, 0, 2, 3, 6. This model inherently assumes that state-to-state probabilities are not conditional, and that the probability of the score differential being 5 or 9 – both technically possible under the new rules – is negligible.

We will let be the transition matrix for the receiving team’s first offensive possession, be the receiving team’s first defensive possession, be every subsequent receiving team offensive drive, and be every subsequent receiving team defensive drive. The first row/column of each matrix represents the receiving team at a -6 scoring difference, and so on until the last row/column is the receiving team at a +6 scoring difference.

The matrices have the following forms:

Momentum Part 5 - Series Level Analysis

This is the final part of my series on momentum in a football game. Is momentum a causative property that a team can gain or lose, or is it only something our minds project to explain streaks of outcomes that don't alternate as much as we expect? It's been a couple months since I began this series, so as a refresher, here is what I've looked at so far:

Part 1 examined the possibility that momentum exists by measuring whether teams that obtain the ball in momentum-swinging ways go on to score more frequently than teams that obtained the ball by regular means.

Part 2 looked at whether teams that gained possession following momentous plays went on to win more often than we would otherwise expect.

Part 3 focused on drive success following a turnover on downs, which is often cited by coaches and analysts as a reason not to go by the numbers when making strategic decisions.

Part 4 applied a different method of examining momentum by using the runs test so see the degree to which team performance is streakier than random, independent trials.

In this part, I'll apply the runs test at the series level, to see if teams convert first downs (or fail to convert them) more consecutively than random independence would suggest. But first, I'll tie up some loose ends left hanging from part 4. Specifically, I'll redo the play-level runs test to eliminate potential confusion caused by a team with disparate performance from their offensive and defensive squads.

The Value of a Timeout - Part 2

In the first part of this article, I made a rough first approximation of the value of a timeout. Using a selected subsample of 2nd half situations, it appeared that a timeout's value was on the order of magnitude of .05 Win Probability (WP). In other words, if a team with 3 timeouts had a .70 WP, another identical team in the same situation but with only 2 timeouts would have about a .65 WP.

In this part, I'll apply a more rigorous analysis and get a better approximation. We'll also be able to repeat the methodology and build a generalized model of timeout values for any combination of score, time, and field position.

Methodology

For my purposes here, I used a logit regression. (Do not try to build a general WP model using logit regression. It won't work. The sport is too complex to capture the interactions properly.) Logit regression is suitable in this exercise because we're only going to look at regions of the game with fairly linear WP curves. I'm also only interested in the coefficient of the timeout variables, the relative values of timeout states, and not the full prediction of the model.

I specified the model with winning {0,1} as the outcome variable, and with yard line, score difference, time remaining, and timeouts for the offense and defense as predictors. The sample was restricted to 1st downs in the 3rd quarter near midfield, with the offense ahead by 0 to 7 points.

Results

Seahawks Should Not Have Scored

I'm not going to touch Sean Payton's late field goal attempt, down eight, into the wind, with only one timeout remaining. After the miss, on the ensuing Seahawks drive, Marshawn Lynch tore down the left sideline and it appeared he was let into the end zone by the Saints. The question is, did the Saints make the right decision letting him score once the first down was achieved - and should Lynch have gone down at the 1-yard line? The answer to both of these will always be the same, and in this case, it's yes.

Lynch scored with 2:48 remaining in the game. Up by 15, kicking off to the Saints, the Seahawks had about a 99% chance of winning. The Saints would need to score, recover an onside kick, and score again just to tie.

Update: Lynch actually scored with 2:40 remaining, not 2:48 as originally listed in the play-by-play. That means that the Seahawks could have milked the entire clock as listed at the very bottom of this article.

But, what if Lynch had gone down at the one? Using the time calculator, we can see how much time the Saints would have had following a chip shot (99.7%) field goal from the 1-yard line. The Saints had no timeouts left, so we are looking at the blue line.

Chargers Courageous Call & Playoff-Clinching Drive

Despite the controversy surrounding an illegal defense on the Chiefs' missed field goal at the end of regulation, the San Diego Chargers defied odds and clinched a postseason berth on Sunday. In overtime, Philip Rivers orchestrated a 17-play, nine-and-a-half minute field goal drive to start the extra quarter that ultimately sealed their win. The length of the drive, in this case, is just as important as the outcome as San Diego could advance with either a win or a tie.

Using our Markov model, let's take a look at the drive. Keep in mind, the model is best used for a standard drive when time and score differential would not greatly affect decision-making or play-calling. Since this was the opening drive of overtime, those standards will predominantly hold true, although not perfectly given the leverage of the situation.

PIT Should Not Have Scored the TD

With the score tied and 1:51 to play, PIT had a 1st and 10 on the GB 17 yard line. In many circumstances a team can run down the clock and kick a short FG to win the game. PIT was near the 'Field Goal Choke Hold' zone, when it's better for the offense not to score a TD and better for the defense to allow a TD. But fortunately for GB, they had all 3 of their timeouts, and could be assured of getting the ball back with 1:27 to play if they made a stop and forced the FG. So with 3 timeouts remaining, the numbers say it never makes sense for a defense to intentionally allow the TD.

But GB jumped offside on the FG attempt, and gave PIT a 1st down and goal from the 5 with 1:35 to play. Now GB had only 1 timeout left, and it would have certainly made sense for PIT to refrain from scoring the TD, burn time off the clock, and kick an easy FG for the win.

The chart below illustrates when a defense would prefer to allow a TD. The black diamond represents the state of the game at the 1st and goal mark assuming PIT does not score a TD. The black line shows the win probability of the defense if they allow the TD.

Seahawks Stumble, Should Have Allowed TD

In one of the most anticipated games of the week, the San Francisco 49ers took over down 17-16 to the Seattle Seahawks with 6:20 remaining. After a huge Frank Gore 51-yard run, the Niners lined up for a 1st-and-Goal from the 7-yard line with 2:39 remaining. Seattle had no timeouts remaining. Should the Seahawks have tried to intentionally allow the Niners to score a touchdown? Let's look at Brian's graph for this situation in his intentional TD study:

Was Belichick Right to Take the Wind in OT?

I was surprised when Bill Belichick chose to take the second possession (and risk no possessions) in OT against Peyton Manning and a team that had scored 31 points in four quarters. Although the new OT format mitigates the advantage of the team with first possession, it's still there to the tune of about 56% to 44%.

The advantage of wind must have felt fairly strong to Belichick. His team captains thought he was crazy. At the time, it was impossible to tell from the comfort of my sofa how bad the wind was, but I was curious if we could see the effect statistically.

Is the Revolution Over? Did We Win?

"The Revolution Was Televised. The fourth down revolution is over. Going for it won."

-Mike Tanier, Sports on Earth

Is Mike right? Did going for it really win? Mike makes a the case, and cites several promising examples of unconventional 4th down decisions from one Sunday afternoon earlier this season:

"-The Lions going for it on 4th-and-goal from the two-yard line, early in their win over the Cowboys.
-The Dolphins going for it on 4th-and-1 from the Patriots' 38-yard line, in the second quarter.
-The Patriots going for it on 4th-and-4 from the Dolphins' 34-yard line, while leading by three points in the fourth quarter.
-The Bengals going for it on 4th-and-inches from the 1-yard line, while leading 14-0 against the Jets.
-The Broncos scoring a 4th-and-goal touchdown to tie the game at 21 against the Redskins, in the third quarter.
-The Packers converting a 4th-and-3 from their own 42-yard line, setting up a touchdown to increase their lead to 31-17."

I think Mike is right to point out some very interesting cases where coaches are making some notable decisions, but the revolution is far from complete. I would suggest that an avalanche is the better analogy than revolution. One day there may be an avalanche of aggressive 4th down decisions, but right now we're only seeing a few rocks trickle down the mountainside. It's not that there haven't been bold examples of enlightenment. It's just that there are so many opportunities that coaches have spurned.

Which Teams Should Abandon the Run?

Yeah, yeah, yeah. It's a passing league. We got it. And still, according to the numbers, teams aren't passing enough. In the cases of some teams, it's painfully obvious that they should be passing more and running less. As a Ravens fan, I watched another game where nearly every run was simply a wasted down. Most of their paltry positive rushing yards seem to come from trash draw plays on long distances to gain, intended to mitigate very poor field position prior to a punt. It's like they're playing with two or three downs when everyone else gets four.

I wonder if, at some point, when an offense is so much better at passing than running, should it abandon the run almost altogether. On top of the general imbalance in the league, some teams are just throwing away downs when calling conventional run plays. Of course, running and passing generally play off of each other in a game-theory sense. To be successful, passing needs the threat of running, and vice versa. But sometimes, the cost of running is so high for some offenses, that it would be worth the trade-off to forfeit the unpredictability and just pass nearly every down.

It sounds crazy, but take a look at the Expected Points Added per play so far this season (through the 1pm games on Sunday 10/13). The right-most column is the pass-run split. The bigger that number, the greater the imbalance. Pay particular attention to the teams highlighted in red:

Examining the Value of Coaches' Challenges

Kevin Meers is the Co-President of the Harvard Sports Analysis Collective. He is a senior majoring in economics with a statistics minor, and has spent the past two years or so as an analytics intern in the NFL. He is currently writing his thesis on game theory in the NFL, and probably puts too much thought into how the perfect fantasy football league would be structured.

The coach’s challenge is an important yet poorly understood part of the NFL. We know challenges are an asset, but past that, we do not have a good understanding of what makes a good challenge or if coaches are actually skilled at challenging plays. This post takes a step towards better understanding those questions by examining the value of the possible game states that stem from challenged plays.

To value challenges, we must understand how challenges change the game’s current state. When a play is challenged, the current game state must transition into one of two new game states: one where the challenged play is reversed, the other where it is upheld. These potential game states are the key to valuing challenges.

Let’s look at a concrete example from last season. With two minutes and two seconds left in the fourth quarter in their week ten matchup, Atlanta had first and goal on New Orleans’ ten-yard line. Matt Ryan completed a pass to Harry Douglas, who was ruled down at the Saints’ one-yard line… only Douglas appeared to fumble as he went to the ground, with the Saints recovering the ball for a potential touchback. When New Orleans challenged the ruling on the field, the game could have transitioned into two possible game states: Atlanta’s ball with second and goal on the one, or New Orleans’ ball with first and ten on their own 20 yard line. If the Saints lost the challenge, they would have a Win Probability (WP) of 0.28, but if they won, their WP would jump to 0.88. This potential WP added, which I refer to as “leverage,” is key to valuing challenges. Mathematically, I define leverage as:

Jason Garrett Should Have Punted

by Matt Meiselman

Matt has been helped me crunch some numbers this off-season. He is graduate of the University of Maryland with a degree in broadcast journalism. He's originally from New Jersey, but loves New York sports. Matt aspires to work in sports media and has a passion for sports statistics. 

We're trying hard not to be broken records on 4th down decisions, but this one is special. -BB

Jason Garrett is not typically regarded as a savvy 4th down decision maker, and his clock management skills aren’t the greatest either. Garrett consistently makes blunders in way too many of his strategic decisions. But even after all the mistakes he’s made as the Cowboys head coach, he somehow found a way to top every one of them with a single mistake in Week 2 against the Chiefs.

With 3:50 to play in the 4th quarter, Dallas trailed Kansas City 17-13. It was the Cowboys’ ball, but they faced a 4th and 10 from the Chiefs 35. They held all three of their timeouts and undoubtedly still had a decent opportunity to come away with a win. Garrett had a decision to make: should he go for it? Punt? Or a kick a field goal?

Based on the 4th down calculator, the Cowboys chose the worst of the three options, and it wasn’t even close.

Saints Slip Up, Still Win

Sean Payton is back in the decision-maker chair for the Saints after his one-year sabbatical, and he is typically known as one of the more analytical coaches. The Saints pass more often than other teams, they go for it on fourth down more often, and even try the occasional surprise onside kick.

On Sunday, Payton faced a couple of fourth down decisions, testing his analytical mind. On their second drive, the Saints ran the ball on 4th-and-1 from their own 47-yard line -- a sizable gamble to the traditional football mind. Mark Ingram was stuffed and the Falcons would turn good starting field position into a field goal. This was definitely the right decision, despite the outcome (+0.84 EP going for it vs -0.30 EP with a punt).

The much bigger decision, though, came with 3:30 left in the game up three. The Saints faced a 4th-and-2 from the 4-yard line after a 5-yard completion to Lance Moore. Drew Brees lined up to attempt a 4th-down conversion but Payton ultimately decided to take a timeout to think things over. After the timeout, the Saints came out in field goal formation, made the chip shot and went up 23-17.

So, what do the numbers say?

Win Probability Forfeited 2012

by Matt Meiselman

Matt has been helping me crunch some 4th down numbers this off-season. He is a senior at the University of Maryland studying broadcast journalism. He's originally from New Jersey, but loves New York sports. Matt aspires to work in sports media and has a passion for sports statistics. -BB

Fourth down decision making is one of the most controversial aspects of coaching in the NFL. Act too aggressively and miss? You get blamed for a loss. Act too timid? The fans will be calling for your head. Most NFL coaches are operating with the mindset that whatever puts their team (and their job) at the least amount of risk is the right choice to make. This has been, and will always be, the wrong way to try to win a football game.

In last year's article on this subject, Brian talked about how coaches aren’t just saving wins if they’re more aggressive; they are simultaneously forfeiting wins by being too meek. In 2011, the average team forfeited .65 wins for the year on 4th down decisions alone. The NFL has started to become more risk-friendly instead of risk-averse, and you’d expect that with more innovative minds in the game, like Bill Belicheck and Jim Harbaugh, the league would be trending towards more optimum game management. This was not the case in 2012.

During the 2012 season, the average team forfeited .73 wins, a significant increase from the year before. The average win probability forfeited per opportunity also rose, jumping from 1.6% of a win to 1.9% of a win. Below are the calculations for each team: