Pre-Season Predictions Are Worthless

I recently came across online discussions regarding pre-season predictions of win totals for each team in the NFL. In this post I'll show why you shouldn't put much faith in any of the predictions you read before the first snap of the 2007 regular season.

For comparison purposes, I'll use Football Outsiders' predictions and the Vegas over-under lines for the last two NFL seasons as representative statistical and consensus predictions. As it turns out, neither fare very well in their predictions.

I'll judge the accuracy of predictions by mean absolute error (MAE). This is the average of the absolute value of the error. If the MAE is 3.5, it would mean the predictions were off by an average of 3.5 games in either direction. The smaller the MAE, the better the prediction.

For the 2005 and 2006 NFL seasons the accuracies for Football Outsiders and Vegas betting lines are listed in the table below.






















YearFOVegas
20053.03.0
20062.32.3
Avg2.62.6


Over the past two years, the Football Outsiders predictions are no better than the consensus Vegas lines. They are wrong by an average 2.6 wins for each team. Although the NFL is difficult to predict, 2.6 games is quite a bit in a 16 game season.

How do we judge whether 2.6 games is a good or bad prediction? To judge if predictions are worth anything, we should compare them to obvious knowledge. I compared the FO and Vegas predictions to two sets of obvious predictions. The first is if I mindlessly predicted 8 wins for every team. The second is just using a regression of last year's wins. The resulting comparison of average errors is listed below.































YearFOVegas8 WinsLast Year
20053.03.03.02.9
20062.32.32.12.2
Avg2.62.62.52.5


The average errors for '05 and '06 was 2.5 games for both the mindless 8-game predictions and last year's records. Both "obvious" methods actually do slightly better than the expert predictions.

Pre-season predictions are completely worthless, at least those of Football Outsiders and Las Vegas. In fact, they're worse than worthless.

One final note--If each division is ranked by their current Vegas over-under lines, the results are virtually identical to last year's final standings. The only exceptions are two ties. If you want to know how to take advantage of bad expert predictions, read this.


Hat Tip: Some data for this analysis came from Football Prediction Network. Original idea stemmed from the Sports Economist via Sabermetric Research.

End note: The regression of season win totals based on previous year wins yielded the following result (n=96):
Next Yr Wins = 5.7 + 0.29 * Last Yr Wins.
r-squared=0.12. Significance for Last Yr Wins was p=0.00.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

11 Responses to “Pre-Season Predictions Are Worthless”

  1. Tarr says:

    Is there any discernable tendency if we look at the predictions where Vegas/FO/8-8/regression to the mean disagree strongly with one another?

    Let me give an example. I have no intention of wagering on team wins. But if I were to place such a wager, I would look for teams where:
    - The Vegas over/under and the FO prediction differed by more than 2 games.
    - The Vegas over/under was far off from 8-8, and the FO prediction was much closer.

    Is this strategy any better, statistically, than taking the over on teams with o/u<6, or the under on teams with o/u>10?

  2. Brian Burke says:

    The only pattern I recognize is that expert picks tend to go out on a limb. They need to validate their existence, so if they pick teams to win within a range of 6-10 games, no one would bother paying attention. But the smartest prediction strategy is to go as short out on a limb as possible (like the 8-win strategy).

    I'd put money on teams that Vegas has as outliers: 5 or less wins and 10 or more wins. I'd bet the under on the good teams and the over on the bad teams. But for now that's a hunch.

  3. Chip says:

    FO's projections are not great, but not worthless either.

    I looked at the Pearson rank correlation for the FO's and Vegas' projections as well as last year's record:

    2006 2005
    FO 43% 33%
    Vegas 45% 20%
    Last Year's 30% 20%

  4. Brian Burke says:

    Chip-

    By "last year's record" I mean a simple regression based on last year's record, not just using a team's previous wins as the projection for next year. See the end note for the formula.

    What this does is simply apply a regression to the mean to last year's wins. So a prediction for a 13-win team would be about 11 wins, and a prediction for a 2 win team would be about 5 wins.

  5. Derek says:

    I think there are other ways to look at it besides mean absolute error. By MAE, yeah, they're no good. Correlation coefficient is another way to look at it. I looked at it another way (as I know you've read) and found that 17 out of the 32 predictions were within 2 games of being correct, 10 of those being within a game. That's not bad actually.

    And maybe it's not the exact win totals that matter as much as the projected division rankings. Or maybe you could see how often the win total range with the highest projected probability turns out to be the correct range.

  6. Brian Burke says:

    Derek, I agree MAE is only one way to look at it. But I disagree that 17/32 within 2 games, and 10/32 within 1 game, is any good.

    I ran the same comparison for the "mindless" 8-win prediction method (which I guess should be called the "null prediction").

    Over the past 2 years, it was within 2 games an average of 16.5 times, and within 1 game an average of 10 times.

    For 2006 it was within 2 games 16 times and within 1 game 10 times.

    By your standard, FO adds a single game of valid prognostication over a 256-game season. It took dozens of FO "game charters" and ultimately thousands of man-hours of analysis to come up with their predictions. That's not so good. In fact, some could call it embarrassing.

    But I do agree with your point that it's really the division rankings that matter.

    By the way, thanks for the great original post on this matter.

    Also, I think everyone will like tomorrow's post. (Tarr-you're on to something.)

  7. Anonymous says:

    Here's a similar discussion on the FO site:
    http://footballoutsiders.com/discussion/viewtopic.php?t=146
    I did find the FO predictions to be better (in 2006) when ignoring the teams that had muddled predictions, but only marginally better in 2005. Not a whole lot of data to use.

  8. Brian Burke says:

    Miles-I saw that post, but I don't understand what you mean by muddled predictions.

  9. Derek says:

    Thanks for running that analysis on the 8-win predictions. I didn't really feel like it to be honest. I'm surprised how little value the FO predictions add. But I think this is why it's important to publish your error rates and accuracy metrics. If you're trumpeting what you got right, you need to be forthright about what you got wrong. It's not good business (and clearly FO has built a reputation for good preseason predictions if TMQ is using them), but it's good science.

    I was curious about your take on the . These seem to be adding error on top of error (the preseason VOA projections and then adjusting for strength of schedule based on those projections). Honestly, I can't imagine Miami having the worst offense in the league. That's my own baggage, I know, but I honestly don't understand why Trent Green is projected to do so poorly.

    And by muddled predictions, I imagine he means New Orleans.

  10. Unknown says:

    yeah -- sorry muddled projections wasn't very clear. FO projects the probability of wins by bucket. Some of the teams are very clearly projected to be very good or bad -- Philly ~90% in the 9-10 plus 11+ buckets; KC ~70% in the 0-4 plus 5-6 buckets. Some team's projections are ~equally spread through all buckets, like Minn -- 10% in the 0-4 and 11+ and ~25% in the others. If you throw out these (the muddled ones), the projections look a bit better. This likely also explains why FO keeps winning the Slate competition -- its good at picking the division winner, but very mediocre at picking everyone.

  11. Marver says:

    Do you think it makes more sense to regress teams that have established winning continuity less than teams that seem to have more volatile records?

    Lets say you and I are going to bet one another a dollar per game in which we're off on our predictions ie. if I predict the Panthers to win 11 games and they win 9, I owe you $2. We make our picks in separate rooms at the same time to avoid any betting strategy based upon what we know our opponent would do.

    What would be your strategy? Your data here seems to suggest that you'd choose mindless 8-8 predictions across the board since its error is the same as the "experts" (ha!), and you'd rather not spend the time it takes trying to come up with an algorithm to produce future wins if it'll be no better. But would you really bet like that?

    I have to believe that picking the Colts to win 9 games, everyone else to win 8, and the Raiders to win 7 would probably net me $2 against that system.

Leave a Reply

Note: Only a member of this blog may post a comment.