Super Bowl Game Probability and Voodoo Analysis

The game probability for the Super Bowl is up at the New York Times' Fifth Down.

This week I talk about momentum and 'voodoo' analysis:

A boulder rolls down a hill and gains momentum. A spark sets a fire, and soon it has built into a blaze. The rains come and soon the river is rushing over its banks. Momentum is everywhere in nature, but applying it to abstractions like team win-loss records in a relatively small sample of football games is what I call voodoo analysis.

Voodoo analysis is the application of apparently intuitive patterns beyond their natural settings. A football team is not a boulder rolling down a hill. It’s not a river bursting through a damn. It’s not a spreading fire. Our brains are continuously looking for patterns like these, and often see them even when they’re not there. That’s why we’re better off taking a disciplined look at the numbers from the full season.

Super Bowl Statistical Matchups

Here is your one-stop shop for all things advanced stats for SB 47. Team efficiency stats. Advanced team stats. Top individual performers from each team.

Clutch Persistence?

I recently wrote about clutch QB performance in the post-season. This post will take a look at clutch QB performance in the regular season and how well it persists from year to year. The approach is to compare how well a QB performs in high-leverage situations to how well he performs without respect to leverage. To do this, we can compare his Expected Points Added (EPA) to his Win Probability Added (WPA). This involves computing an "expected" WPA for each QB's season based on his EPA. The difference between a QB's actual WPA and his expected WPA could be considered his "clutch-WPA."

Here is a graphical depiction of what I'm talking about. This chart is from a 2010 article on clutch play. The vertical distance between a QB's expected WPA and his actual WPA, shown as the red line below, is clutch-WPA.

For this analysis, I used per-play metrics: EPA per play, WPA per play, and clutch-WPA per play. Only QB seasons with greater than 200 pass attempts were included.

To estimate "persistence" I measured the year-to-year correlation in our three variables of interest. The idea is that the stronger the correlation, the more persistent the measure is as a quality of that QB. If there is no year-to-year correlation, then the variable may only be random.

The table below lists QB year-pairs and their correlation in our three variables of interest. The 'n' column lists the number of year-pairs in the analysis. For example, the  1 - 2 row represents the 90 cases of first and second seasons each QB appears in the database. The database begins in 2000, so year 1 does not necessarily represent a QB's rookie season or his initial season with 200 attempts. (Note that both seasons in each year pair had > 200 pass attempts.)

Clubs Stay Conservative In No Man's Land

Initially, I was surprised Bill Belichick elected to punt last Sunday on a 4th-and-8 from Baltimore's 34-yard-line. Belichick's Patriots were up by six, and Belichick of all coaches should understand a six-point lead in today's NFL is far from safe. The decision was far from a no-brainer, as Keith sketched out last week.

But those calculations use generic odds; Belichick had his staff's play calling and Tom Brady's arm on his side. As well as his defense had played in the first half, it was still the 29th-ranked squad by efficiency entering the game. Joe Flacco and the Ravens offense, although mediocre overall, showed an ability to pick up large plots of yardage in just a few plays. The difference between handing the Ravens the ball at their 34 or around their 10 (Baltimore fair caught the eventual punt at the 13) seemed worth risking for the Patriots in order to maintain possession.

Of course, the Ravens proceeded to drive 87 yards as Joe Flacco dissected New England with a 13-play no-huddle masterpiece of a drive. Baltimore took a 14-13 lead and wouldn't relinquish it.

Belichick is a risk-taker, indeed -- even if he softens up between now and his retirement (assuming whatever form he takes on this earth is subject to human aging), his famous decision to go for it on a 4th-and-2 in his own territory against Indianapolis in 2009 cemented that legacy. If Belichick wouldn't try in that situation -- one we suggest is a toss-up, slightly favoring a conversion attempt -- who would?

Unsurprisingly, attempting to convert a 4th-and-8 between the 30 and 40 outside of the fourth quarter isn't just outside the limits of Belichick's courage. NFL teams faced the situation 15 times this season (including playoffs) before Belichick faced it last Sunday, and 15 times the teams kicked -- four punts and 11 field goals, with just six successful.

4th-and-8 happens to be a convenient example -- teams at least tried once on everything else from 4th-and-1 through 4th-and-11. But coaches shy away from attempting any fourth down longer than one yard, and anything longer than three is treated like the plague (looking at just the first three quarters to try and eliminate desperation attempts).

Powered by Tableau

Coaches tried for the first down over 75 percent of the time on 4th-and-1, but an extra yard scared away half those brave souls. For some reason, teams almost never attempted with between six and nine yards to go, but were more willing to risk the fourth down attempt (although I would suggest punting is a risk as well, as the Patriots found out) with 10 or 11 to go.

Here are the plays separated out into individual blocks, with wide blocks representing successes and narrow ones representing failures. You can mouse over the individual blocks for a description of the play:

Who's the Clutchiest Post-Season QB?

What does clutch even mean? To me clutch means someone who has over-performed his typical expected level of performance in high-leverage situations. A great QB who plays equally well in clutch situations as he does in other situations isn't 'clutch' to me. But a guy who raises his game in high-leverage situations can be called clutch.

The ability to over-perform in clutch situations as a persistent skill almost certainly does not exist. (More on this in a future post on year-to-year correlations in general performance levels and clutch performance levels.) But that's not to say that some players' better moments happened to have occurred when things mattered the most. Although clutch as a quality or skill does not exist, clutch as an event certainly does.

Trying to define clutch is a tricky business. It's an arbitrary exercise with no one correct answer. Should we count situations where a team is down one score or two? With 5 minutes left to play? 4 minutes left? The final 10 minutes of a game?

My solution is to compare a player's Expected Points Added (EPA) with his Win Probability Added (WPA). EPA measures total production without respect to time and score. In contrast, WPA is heavily weighted by game situation. Players whose WPA exceeds what we'd expect based on their EPA could be thought of as clutch, and players whose WPA is below what we'd expect could be considered anti-clutch.

Patriots Primarily Punt on Fourth Down

Bill Belichick is known for being one of the greatest football minds in NFL history. He's also known for being one of the "riskiest" play-callers -- riskiest in quotes to emphasize that he actually plays to the odds rather than most of the conservative football minds. Down 28-13 in the AFC Championship game, avid Patriots fan Bill Simmons put it best: "Ravens playing to win, Pats playing not to lose."

Belichick faced eight fourth downs in the game against the Ravens, seven of which were legitimate questions for the best course of action: Go for it, punt or kick the field goal. Whereas we would normally expect Belichick to be aggressive, he seemed more reserved in his decision-making. There are a ton of factors that could explain his passive play-calling. For example, it was extremely windy making field goals more difficult and maybe Belichick did not have faith that Joe Flacco could sustain a 90-yard drive due to the Ravens boom-or-bust offense (the Ravens ended up with three scoring drives of 10+ plays including a 90-yarder and 87-yarder).

Let's look at each fourth down decision, starting with the very first. Remember in the beginning of the game, especially if the score is fairly close, we should look at expected points, but as score and time become bigger factors, we will switch to win probability. Also, note that these are league baselines. The fact that the Patriots offense is No. 1 in the league and far above league average would indicate a higher success rate on 4th-down conversion attempts.

Feature Enhancement: Time Calculator

The Time Calculator is a tool that will estimate the time remaining in a game that a trailing defense can expect to get the ball back if they force a stop. It considers the current time and timeouts remaining while factoring in stoppage from the two minute warning and change of possession.

The previous version of the Time Calculator could only base its estimate beginning with the time of the first down snap of a series. For the vast majority of situations that's ok, because offenses will typically only run plays that avoid stopping the clock--runs that stay in bounds. But sometimes there is a stoppage, due to either an incomplete pass,a runner going out of bounds, penalty, or other reason.

The old calculator could account for an unexpected stoppage if you add a notional timeout to the game state. For example, say the defense began the series with 1 timeout, then used it following 1st down, and there was an unexpected stoppage after second down. This scenario would be no different than if the defense began the series with 2 timeouts rather than their actual 1.

Still, it would be easier and more straightforward to make the calculator work for any down. Now you can enter the time at the snap of any down in a series along with the number of timeouts remaining, and the calculator will estimate the time after the change of possession.

All the other options remain the same: the average duration of each play, the game-clock duration between plays, and whether the defense would prefer to trade away some time on the clock to preserve a timeout for use on offense.

Try it out. The Advanced NFL Stats Time Calculator.

Against Colin Kaepernick, Pick Your Running Poison

Questions about Colin Kaepernick's ability to lead the 49ers in the playoffs were brushed aside with alacrity in last Saturday's emphatic victory over Green Bay. Kaepernick dominated the game from start to finish, landing blow after blow with his legs until a 56-yard touchdown run sent the Packers packing back to Wisconsin and broke the quarterback playoff rushing record in the process.

As Brian noted at the New York Times's fifth down, Kaepernick is the main reason San Francisco gives a substantial 4.5 points at the Las Vegas books despite playing at top seeded Atlanta in the NFC Championship. The 49ers' solid conventional running game and excellent defense would have made them a formidable matchup for the ever-questioned Falcons with Alex Smith under center. Kaepernick's dynamism in the running game makes San Francisco a nightmare.

The zone read and other designed quarterback running plays -- such as those out of the pistol formation -- have used Kaepernick's speed in a wildly successful fashion. Including the victory against Green Bay, Kaepernick has used designed runs 35 times and picked up 351 yards and 21 expected points added, an absurd 0.6 expected points added per play. Marshawn Lynch had 19.6 expected points added this season on 315 rushing attempts.

Kaepernick piles on the big plays. Only 41.4 percent of rushing plays have gone for a positive EPA and only 9.2 percent have gone for more than plus-1.0 EPA. Among just Kaepernick's designed runs, 57.1 percent have been positive by EPA. Most impressively, 11 of his 35 (31.4 percent) have been worth at least a full expected point added; four of the 35 (11.4 percent) have been worth more than three.

Powered by Tableau

But, even if the Falcons can stop Kaepernick out of the pistol or zone read or other designed runs -- and they haven't stopped quarterbacks running yet this year -- they still have to deal with his legs after he drops back to pass.

Kaepernick's passing ability is dangerous enough -- including the playoffs, his 6.7 AYPA leads the league. Add in his legs and it makes truly covering the field nearly impossible. Kaepernick has busted out of the pocket 29 times for 244 yards, picking up 8.4 yards and 0.39 EPA per scramble.