One of the most random events in football is the interception. We can say that not just because of tipped balls or freak gusts of wind, but because we know that interception rates for teams and for individual quarterbacks varies widely from season to season. They also vary greatly from one half of the season to the next, which suggests that more is at play than player ability. In a recent post I estimated the proportion of team win-loss records that can be attributed to sample error in an effort to demonstrate the technique.
I looked at the interception rates (int per attempt) for all ‘qualified’ NFL quarterbacks since from 2002-2009. To qualify, a QB must have thrown a certain proportion of his team’s pass attempts. The overall average interception rate is 2.9%. The standard deviation, from QB to QB, is 0.94%, which means about 2/3 of the QBs will have interception rates that fall somewhere inside 2% and 4%.
To calculate the variance due to randomness, I couldn’t directly use the formula from the binomial distribution. Unlike team seasons, which all have 16 games, QBs have varying numbers of attempts during a season. Instead, I used a random simulation to estimate the random variance. I started with the premise that all interceptions were completely random. What if every quarterback’s pass attempts each had a 2.9% chance of being intercepted, regardless of who he is or who the opponent is. What would the distribution look like then? How different would this purely random distribution be from the actual distribution we observe in the real NFL?
This is something similar to a study Chase Stuart did at last year at Footballguys. He created 25 notional QBs, each with an identical interception rate. He demonstrated that after 500 attempts each, there could be a wide range (from 9 to 22) in total interceptions due to sample error alone.
I did things differently. I re-ran the past eight NFL seasons worth of pass attempts as a random simulation. Each real QB had the same number of attempts as he did in the real seasons, but his simulated number of interceptions was a purely random function based on the baseline 2.9% interception rate. After simulating hundreds of seasons, I looked at the resulting distributions. The mean interception rate was obviously the same as the actual rate at 2.9%, but the standard deviation was narrower, as we’d expect. The purely random simulation had an average 0.85% SD compared to a 0.94% SD for the actual observed seasons.
The proportion of randomness of interceptions is therefore:
In other words, the variance of a QB’s season-long interception rate could be up to 81% due to random sample error and as low as 19% due to persistent player ability. But variance is an abstract statistical concept. It may be more intuitive to think in terms of correlations. The correlation between a QB’s actual interception rate during a season and his underlying true ability to avoid interceptions is sqrt(0.19) = 0.43.
In the graph below, you can see what I'm talking about. If interceptions were purely random, we'd expect to see a binomial distribution centered near an average rate of 2.9%. The red line is just one set of 8 seasons of randomly generated interception rates. The blue series represents the actual distribution of interception rates over the past 8 seasons. They're not much different. The random distribution is taller and narrower. The fact that the actual distribution is wider and flatter indicates the degree of skill involved.
We can see from the graph that some coaches may tend to have a boiling point--a limit beyond which they will not tolerate any more interceptions. Once a QB gets to just under a 4% interception rate, he'll be replaced. If true, this would create a selection bias in the actual distribution. Without this bias, the underlying actual distribution would be slightly wider, increasing its variance.
Certainly interceptions are due to skill at some level. If you put me in at safety on an NFL team, chances are I'm not going to reel in a single pick. Or if you put me in a quarterback, I'd throw 10 interceptions per game. Obviously, speed, experience, and skill all play a part. The thing is, average Joes like me aren't out there playing in the NFL. The skill levels of the participants are relatively equal, even for the very best and very worst of active players, and that's part of why the noise of randomness is so large compared the signal of skill.
So:
Total = 0.94^2
Randomness = 0.85^2
Subtracting, we get
Talent = 0.48^2
So the SD due to randomness is almost twice the SD due to talent! Do I have that right? I wouldn't have expected that.
Excellent. I love seeing variances.
We also know that interception rates per attempt are lower as the offense's lead increases and higher as the offense falls farther behind. Since some teams are better than others, and thus QBs find themselves in different average lead/deficit situations, I'd expect if you accounted for the score you'd find interceptions to be even more random than suggested above.
We also know that interception rates per attempt are lower as the offense's lead increases and higher as the offense falls farther behind. Since some teams are better than others, and thus QBs find themselves in different average lead/deficit situations...
I looked at this once and for some QBs the effect of being ahead/behind seemed very marked. This is a good observation worth remembering when judging QB performance.
@JimG
Well when the difference on total interceptions is already 60% random chance, then I would expect any marked difference on a further decreased data set, is very likely just sample size problems.
But it would be interesting to check if you controlled for game situation, if the total variance would be even better explained by random chance then anything else.
And as a topping you could also consider the obvious feedback loop QB throws INT => QB gets behind => QB throws riskier => QB throws more INTs
I would not be surprised if that set of data explained over 90% of the total variance.
Don't interceptions correlate very closely to game situations when they are thrown? It seems, for example, that very few interceptions happen on a 1st and 10 in the first quarter with the score tied, because the quarterback sees a low reward for trying to force a high risk pass. Conversely, interceptions seem much more common in the 4th quarter when a quarterback's team is losing and it's 4th down and he needs to try to force a miracle play to keep a drive alive. I've always felt that's the proper cause-effect relationship of quarterbacks with a lot of interceptions playing on losing teams. i.e. It's often the losses that cause the interceptions, not the other way around.
Yup. Correlation/causation.
Somewhat off topic, but it is related to interceptions. Brian - Would you be able to run Peyton Manning's EPA and WPA for the New England game yesterday. The Colts message board I am on is arguing whether Peyton cost us the game or whether he was the only reason we were in it in the first place.
My guess is that he still had a positive EPA and WPA for the day, even though all three of those INT's were fairly costly from an EPA/WPA perspective.
Pardon my ignorance in statistics but I am not sure I understand why the distribution illustrates randomness for a quarterback. Why is persistance not the key metric. If you have a perfect binomial distribution but one in which a set of QBs is consistently (statistically speaking) on one half of the curve and an set is consistently on the other half, you have evidence of skill.
With regards to game situation, has this been studied? If so, could someone please direct me to the results?
Thank you.
Michael, I'd assume Peyton would have a positive EPA but I'd think that last pick would be enough to offset any WPA he'd earned in the game, it alone was worth -0.45 WPA.
@JimG Well when the difference on total interceptions is already 60% random chance, then I would expect any marked difference on a further decreased data set, is very likely just sample size problems
I wasn't talking about interceptions but all measures of QB performance, AYA, completion pct., etc.
Sample size is a problem, the more closely you parse situations the smaller the samples become, but that's life in the NFL, only 16 games. Still, if in the great majority of samples the great majority of measures lean in one direction....
Causation is always a problem, but here not such a one as it might seem. You look at the score at the time of the pass attempt. If the QB is behind then it is because falling behind due to bad play that occured when he *wasn't* behind. So his own bad plays that may have caused him to fall behind are excluded. (In fact, if his own bad play put him behind then regression to the mean could cause his "behind" performance to be *better* than his average, as those bad plays of his are excluded from these stats.)
And as a topping you could also consider the obvious feedback loop QB throws INT => QB gets behind => QB throws riskier => QB throws more INTs
Oh, for sure. Chad Pennington is the most accurate passer in NFL history. I watched him on good Jets teams never throw a "bad" pick. Then on bad Jets teams I watched him throw one seemingly "bonehead" bad pick after another trying to come from behind late throwing uphill into a D waiting for it. Same football brain.
My personal opinion is that game situation is so important for QB performance that the quality of the defense playing while the QB is resting on the bench materially affects the QB's passing statistics, by helping keep him ahead or dropping him behind.
A striking "meta" fact about HoF QBs is that they *all* played on good teams, unlike HoF players at all other positions none ever played on a team that lost year-after-year, winning nothing. (Compare HoF running backs!)
So either no bad team ever drafted a QB with HoF ability, as they do at other positions, for some unknown reason, or the stat lines that get QBs into the HoF are *very much* the result of factors other than their own ability, the other players on the team, game situations, coaching, usually being ahead rather than behind because of teammates (including defensive teammates) etc.
Jim, another solution would be HoF quarterbacks are so good they make their teams good regardless of the players around them. I don't think that's entirely correct, but it's possible.
I think the main problem with HoF quarterbacks is a quarterback gets credited with his team's W/L record so if the team is bad the voters won't elect the quarterback regardless of his stats.
Very true. Even the best QBs throw ints when behind. Look at Manning this past weekend. Int rate is a function of a lot of things, and game situation is a big factor. But then again, game outcomes are quite random.
The comment about the feedback loop is right on. QB is no good-->team falls behind-->QB looks even worse.
Jim, another solution would be HoF quarterbacks are so good they make their teams good regardless of the players around them.
That's logically possible and certainly fits with the common "the QB carries the team to the title" mentality. But it doesn't seem credible factually. It doesn't square with the PFR.com finding that the average difference between a starting and backup-level QB is only 2 pts a game ... the "win share" logic that a QB is responsible for about 10% or 12% of a team's results ... or NFL payroll data -- if Peyton and Tom really do it all by themselves, then they are *way* underpaid even for all they are getting now.
I think the main problem with HoF quarterbacks is a quarterback gets credited with his team's W/L record...
That's it. If Tom Brady had been drafted in the 5th round by the Lions in 2000, and been handed the starting job in 2001 when the starter got injured, just as happened in NE, would there be a HoF niche waiting for him? I really, really doubt it.
...so if the team is bad the voters won't elect the quarterback regardless of his stats.
The interesting question is, if Tom had been drafted by Detroit how different would his career stat line be today? After playing with all those lousy players on very bad teams instead of top ones on a superior team I'd guess "very, for the worse".
It's easy for me to imagine that after going in when Batch got hurt, and leading the team to a failure 4-12 season (instead of 2-14), he would've been judged just another low-pick backup who couldn't do the starter's job and been benched then next year for savior top-pick Joey Harrington. We might not even know Tom's name today.
That's my imagination at work, we'll never know of course. But IMHO when Belichick drafted Brady in the 6th round he won the Powerball Lottery -- and so did Brady. That's no slight to Brady. Even great players need a chance to show what they can do. He would've had no chance in Detroit, he couldn't have had a better chance anywhere else than NE. One person's opinion, your mileage may vary.
Even the best QBs throw ints when behind ... The comment about the feedback loop is right on. QB is no good-->team falls behind-->QB looks even worse.
That loop applies even when the QB is good or even great. As you just said, Manning yesterday (or for that matter his critical late pick in last year's Super Bowl.)
That's my point about game situations and the rest of the team significantly affecting a QB's stat lines. Lots of things besides the QB himself can put the QB behind and into that loop that hurts his stats. Game situations are mostly determined by the rest of the team.
@Jim Glass -- Kyle Orton & Jay Cutler make an interesting case study since they switched teams in 2009:
Here are they're Adjust Net Yards per Attempt from pro-football-reference.com
Jay Cutler - 2007: 6.30 (2.9% int)
Jay Cutler - 2008: 6.61 (2.9% int)
Jay Cutler - 2009: 4.81 (4.8% int)
Jay Cutler - 2010: 5.43 (3.7% int)
Kyle Orton - 2007: 5.32 (2.5% int)
Kyle Orton - 2008: 5.35 (2.6% int)
Kyle Orton - 2009: 6.18 (2.2% int)
Kyle Orton - 2010: 7.02 (1.5% int)
It's better to be a QB on the Broncos than the Bears. Each player had a higher int rate on the Bears than the Broncos but Orton throws less ints than Cutler.
I think that we would probably find something simliar with McNabb and Campbell, it's better to be QB for the Eagles and much, much worse to be QB for the Raiders.
The interception feedback loop raises another interesting point. When a team is behind, they clearly need to pass to pick up yards, especially in a situation with a short clock.
Let's visit the idea that most coaches play calling is predictable. This means that defences can defend against a pass and once they are expecting a pass they have reason to jump routes. this results in interceptions.
I offer that there is an parallel to the idea that teams run too often on first down (a predictable play call.) Teams pass too often when behind in a game. (I appreciate that time may be a factor but I feel that too many teams panic early and start pressing sooner than they need to.)
This is an argument for trying the unexpected.
Calling an unexpected play (a run) may pay dividends in a desperate situation. (We saw a run for a first down turn into a touchdown Sunday night in the Eagle game.)
Within a year, interceptions ARE pretty much random. For example, Garrard's THREE INT's a couple of years ago. This was not indicative of his skill level, just of his luck that defenders probably dropped a few. However, I would most DEFINITELY posit that over the course of a career of several THOUSAND pass attempts, one can definitely identify QB's who were better at avoiding INT's, even after adjusting for era.
I don't have the time to run the #'s, but I bet if you compared four/five QB's who played in the same era, and compared their INT rates, you would find that one avoided more INT's and another threw more than the others (comparing rates, not just totals.) You could also, just between those specific players, compare how many they threw while ahead, tied, or behind--then find the average INT rate for being ahead, tied, and behind--then compare which QB(s) were better or worse at avoiding picks.
I guess my basic point is that IN ONE INDIVIDUAL SEASON there will be a lot of random noise, sample size issues, and simple variance/luck. Over a career, those things even out. THAT'S where you find the skill.
A different approach to determine if INTs are random, is to keep track of each qb, and do a boot strap type of analysis on their output. Then one can determine how much spread is due to individual qbs. i.e does a qb spend as much time above 50% point as below, or do qbs tend to say put in the probability distribution?
The analysis above seems to assume a type of random variable, determine that the probability distribution is different than observed, then make conclusions on that difference. Surely if you changed your probability distribution, you would get different results.
Perhaps, you are trying to answer the question, are ints normal?
Looks more like a poisson type distrubtion that is "mostly truncated", as you point out, just below 4.
Regrding Orton and Cutler, I think that's a fine example.
So Minnesota, Favre, highest interception rate meltdown is due randominess? Is not this clear evidence that QB skill is key parameter for interception rate?
Favre's interception rate is a tiny bit *below* league average for his career. He has so many picks because he has so many attempts, and so many memorable ones because he's been in a lot of big games ... and is so flamboyant about it all.
More evidence by anecdote. Using PFR.com's "game finder" to see what team managed to give up the most turnovers and still win, it turned up a good-weather Sept day in '67 when Lombardi's Packers gave 8 turnovers to the Bears (net -7, Vince must've been thrilled), and still won 13-10. Starr threw five interceptions. The previous season, 1966, he'd thrown only 3 in 14 games -- with the same team, players, offense, coach. Stuff happens.
I would be really curious to see a similar study, but looking at defenses instead of quarterbacks. In other words, instead of looking at INT% for Peyton Manning or Chad Henne, look at INT% for "Chicago opponent QB" or "Houston opponent QB."
I echo the earlier post that this distribution does not necessarily indicate randomness of interceptions. I would suspect that Peyton Manning and Tom Brady are reflected 8 (or at least 6) times on this graph on the left hand side, while there are a steady stream of essentially replacement level quarterbacks coming in on the right hand side with an interception rate just above 3%. That would mean that limiting interception rate is a skill that some quarterbacks have that enable them to excel in the league for a long time, while others are merely passable and get run out of the league once their "skill level" starts to move towards 4 ot 5%. Sure, some crappy quarterbacks will have extremely low interception rates some years and some great quarterbacks will have high ones due to sample size, but overall I don't think this disproves a high level of skill involved in interception rate.