Sam Waters is the Managing Editor of the Harvard Sports Analysis Collective. He is a senior economics major with a minor in psychology. Sam has spent the past eight months as an analytics intern for an NFL team. He used to be a Jets fan, but everyone has their limits.
The NFL season might only be nine weeks old, but the league’s playoff picture is already starting to gain clarity. Having a playoff picture that is too clear, too early, might make games a little less exciting, so I thought it would be interesting to see if this year’s playoff qualifiers really are more certain at this point than in past years.
We can start to attack this question using the projections at nfl-forecast.com, which use the team efficiency ratings here at Advanced NFL Stats to estimate each team’s chances of making the playoffs. According to these projections, eight teams currently have a probability of playoff qualification higher than ninety percent. While these teams (like Denver, Kansas City, and Seattle) are virtual locks, we have thirteen teams with a ten percent chance or less of qualifying, leaving us with a pretty polarized distribution of playoff odds after nine weeks of play:
After Week 17, when the field of playoff qualifiers is completely certain, we will have twelve teams with a 100% chance of qualifying and twenty teams with a 0% chance of qualifying. As you can see in the histogram above, we are already moving in that direction. With the fate of 21 teams almost sealed, we are left with a pack of only eleven teams in the middle vying for the four playoff spots that are still up for grabs. If this degree of clarity is normal, we would expect to see approximately the same number of teams gravitating toward either end of the distribution at this point in previous seasons.
If we want to go about comparing the degree of “playoff picture clarity” across years, we have to find a way to measure it. As we approach a perfectly clear playoff picture, every team gets closer to either a zero percent or one hundred percent chance of qualifying, so we can gauge the clarity of the playoff picture in any given week using the spread of the playoff probabilities for all thirty two teams. I’ll use standard deviation as my measure of spread, so in weeks with a higher standard deviation of playoff probabilities, the field of teams that will qualify for the playoffs is clearer. For example, after Week 17, the standard deviation must be 49.1 (12 teams as 100%, 20 teams at 0%). In contrast, if we assumed every team had an equal chance of making the playoffs, the standard deviation would be zero. If we compare the standard deviation of playoff probabilities in Week 9 this year to the standard deviation in other years, we can find out if and when the playoff picture was this clear in previous seasons:
The standard deviation for Week 9 in 2013 is higher than the Week 9 standard deviation for every year except for 2009. That year we reached the same level of clarity that we have now back in Week 7, when Indianapolis, Denver, and New Orleans had playoff probabilities over 99% and five teams already had a 0% chance at the playoffs (rough seasons for Cleveland, Kansas City, Oakland, Detroit, and Tampa Bay). Meanwhile, in 2010, fans had to wait all the way until Week 15 to get a playoff picture as clear as our current one, with the Chargers and Chiefs battling for the AFC West title while the Seahawks and 49ers fought over the NFC West and the Giants and Packers vied for the sixth seed in the NFC.
If we compare the standard deviation for every week of 2013 thus far to the standard deviations for those same weeks over the last five years, we can see that the playoff picture has been clearer than average since Week 5, when NFL forecast started making these projections.
It looks like the likely field of playoff qualifiers has been consistently more clear than usual over the last month, but judging by the 2009 results, this level of clarity is not unprecedented. As an aside, the dip in week six in the graph comes from missing data – in years that tended to have higher standard deviations, nfl-forecast did not publish week six playoff probabilities. I expect that a larger dataset would smooth this dive significantly.
Our original question was whether this year’s playoff picture was more certain at this point than it has been in past seasons. After looking over these data, it appears that the answer is yes - the likely playoff qualifiers are more certain than they have been over the last five years, on average. But this season is not an outlier by any means, and there have certainly been seasons with greater or comparable certainty at this point over the last five years. The situation looked more extreme before this weekend’s games, with major changes to Green Bay (-19.7 percentage points), Dallas (-11.2), Chicago (+23.2), Philadelphia (+17.5), Miami (+14.0), and Tennessee (+13.7) all pushing the standard deviation back down. So with another week in the books, it has actually become less obvious which teams will be playing in January. Not that there’s anything wrong with uncertainty – not knowing who will be in the playoffs means the remaining games will have even higher leverage, making them more exciting for everyone.
Very cool analysis!
That is definitely a neat thing to be thinking about, but it seems like the Shannon Entropy of the projection would be a much more appropriate measure for the clarity of the playoff picture than taking the standard deviation of teams' chances to make the playoffs.
Building on NateTG's comment, I wonder if you could use the Gini coefficient as a way to measure clarity. It's usually calculated in the context of income inequality (how much wealth is concentrated in the hands of the few). But I would think it would work similarly for playoff probabilities. You basically have 1200% of playoff probability (100% each for the 12 teams). How equally is that playoff probability distributed amongst the 32 teams?
Perhaps the two above are simply using fancier terms that I am, but why not just add up the percentages of the top 6 teams in each conference?
I think this is interesting but nfl-forecast exaggerates the odds of good teams making the playoffs and underestimates the odds of bad teams making the playoffs.
The 2009 teams are a good illustration of the overconfidence of the model. Indy and NO of course made the playoffs but Denver crashed and burned despite that 99% chance of making it.
The NBA playoff forecaster on ESPN tries to correct for this kind of overconfidence with the anti-Dennis Green postulate: "Maybe they AREN'T who we think they are" and you have to take that into account.
Technical explanation of the above:
nfl-forecast (as I understand) makes forecasts by using the GWP formula to calculate a probability of each team winning for each game. Then it does a Monte carlo simulations of the season based on random draws for each game and sees who makes the playoffs and repeats 10,000 times (or so). The projected odds of making the playoff for each team is (# of times made playoffs)/10,000 simulations.
But the GWPs aren't exact. They are based on the team's estimated quality level which is measured with error. So when you do a monte carlo simulation you should not only draw errors for the individual games, all the teams should get errors drawn for their quality. This adds a lot more variation in who makes the playoffs because intuitively there are now TWO ways low probability teams can sneak in: (1) as before they can get lucky in a string of simulated games or (2) they are better than their GWP suggests and they win games based on their underlying strength. So when you simulate it out you'll see more teams with low generic GWP/quality get in and more teams with high generic GWP/quality miss out.
Steve, since the GWPs are already regressed for their predictiveness for the rest of the season shouldn't that mitigate the misevaluation error?
There is still a possibility that they are off--either too high or too low--even if they are right on average.
The easiest way to see why this matter is if you try to simulate the playoffs ignoring the error terms. The probability of winning the Super Bowl for a team with a bye is P(win divisional AND win championship AND win SB). If there is no uncertainty about team quality then those are all independent so you can simulate them and multiply.
But if they aren't independent then you can't. One reason they wouldn't be independent is if Baltimore is actually better than we think, so their GWP is higher for the div round, the CC, and the SB, or worse (so they're more likely to be one and done or win and then crash and burn).
You can see some evidence of the overconfidence if you plot the actual outcomes (smooth) vs the predictions from nfl-forecast like here but the small sample size caveat applies:
https://dl.dropboxusercontent.com/u/2174585/proj_vs_realized.jpg
> Perhaps the two above are simply using fancier terms that I am, but why
> not just add up the percentages of the top 6 teams in each conference?
Not so much:
http://en.wikipedia.org/wiki/Shannon_entropy
http://en.wikipedia.org/wiki/Gini_coefficient
The reason that I suggested Shannon entropy is that it measures how much unknown 'information' is left to completely determine the playoffs -- it's pretty much exactly what you'd use to describe how 'unclear' the playoff picture is.
@Steve -- Adding variance to the GWP has less effect than one would think because of the uncertainty of the GWP of the opponents.
For example after 6 weeks, the expected number of wins for the 2009 Broncos out of their 10 remaining games was 7.4 wins with a standard deviation of 1.2 games.
If you re-do the analysis allowing for a SD of 0.10 in GWP for each team, Denver's SD of games won only increases to about 1.5 games. I doubt it would have changed Denver's playoff odds by more than 1%
Occasionally, rare things happen.
@Chris
I think Denver's GWP had a much larger variance than that based on Brian's model.
Anyway, rare things are supposed to happen rarely, not occasionally.
nfl-forecast predicted 110 team's fates were "almost sealed" (<10% or >90% odds as described in the blog) from 2007-2012 in week 8*. Based on the model's own probabilities, it should have only been wrong about only 2 or 3 of those locks.
But it was wrong 7 times which is enough to reject the model (p=0.078) despite the small sample size.
* - and week 9 for 2008 when I couldn't find week 8