I’ve noted before just how bad expert predictions of NFL season outcomes usually are. It appears to be the same in baseball too. This site tracked and graded 42 expert predictions of which teams would be in first place in each division. Although we’re only just past halfway through the baseball season (the rankings are as of July 1st), 90 games into a season is not too soon to check in on how the experts are faring as a group. There’s little reason to expect much improvement in them between now and September. This article will take a look at just how well the experts do as a group. (Does the title kind of give it away?)
The grading system assigns 12 points to each expert for predicting the current division leader, and then assigns a proportional amount for trailing teams based on how many games out of first they are. For example, someone who predicted the Cubs would be leading the NL Central, he’d receive 12 points. If he had picked the 2nd place Cardinals to win the division, he’d get 10. But if he had picked the cellar-dwelling Reds, he’d get only 1 point.
Of the 42 experts tracked, the two best prognosticators each scored 64 points under the system, which seems pretty good because picking all 6 division leaders would add up to 72 points. But with so many experts, a few are bound to end up with very good scores just by luck alone. Do these experts have any real predictive ability, or is it just luck?
If there was a room full of monkeys (or Sports Center analysts) making predictions purely at random, they’d get an average of 39.5 points according to the scoring rules. The experts’ actual average score was 48.6. Things are looking pretty good for the experts so far.
But the experts had a huge advantage over the monkeys. They can read. Using last season’s standings as a basis for this season’s predictions would score 59 points. How does the experts’ average of 48.6 look now?
Below is the distribution of the experts’ actual scores. For example, 3 experts scored up to 33 points, and 5 experts scored between 34 and 38 points, and so on. I’ve also drawn a normal distribution based on the experts’ actual scores for comparison.
But as I noted above, last year’s standings would be the most logical starting point for making this season’s predictions. Since the experts had the benefit of this prior knowledge, even if they had absolutely no analytic or predictive ability, the distribution of scores should be bunched around the 59-point baseline of last year’s standings. Some experts would score higher while others would score lower, due to either luck or true analytic ability. And if they had any real predictive ability, they should usually score at least that well.
The next graph shows the actual distribution of the experts compared the distribution of how it should look by using the information from the prior season. (It would actually be slightly non-normal and asymmetrical because of being bounded at a 72-point maximum.) Remember, this is still monkeys guessing randomly, but this time their starting point is the 59 points based on last season. How do the experts compare to monkeys who can read last September’s standings?
It turns out the experts aren’t even close to the sports-page-reading monkeys. The distribution of prediction scores is about halfway between last season’s baseline and totally random.
My point is that as we approach football’s silly season of predictions, don’t believe anything you read or hear. Don’t put much stock in a favorable prediction for your team, and don’t panic at an unfavorable one.
Tangentially, I’ve learned something about predictions from the traffic of my site. If I do an analysis that turns out to be favorable for a team’s outlook, such as the Jared Allen article, my site gets linked around the Vikings fan community (which I appreciate—don’t get me wrong). And if I do a negative article, such as the one about Favre, people are interested, but there’s not as much excitement.
So if I had ads on my site, I’d be temped to heavily skew my analysis positively, giving every fan a reason for hope. I’d probably go around the league, targeting certain markets with fancy-schmancy math about how awesome their team is going to be. Except for you, Pittsburgh fans. You guys are going to stink this year. Besides, you don't understand math. (That's another technique--create controversy.) My traffic would explode, and I’d make a little money. I get the feeling much of the national sports media probably does just this sort of thing. Everyone is selling hope I suppose, even to poor, stupid Steeler fans.
Hat tip: King Kaufman.
Baseball Experts: Dumber than Monkeys
By
Brian Burke
published on 7/05/2008
in
predictions,
research,
reviews
Subscribe to:
Post Comments (Atom)
So, if the overriding strategy in predicting NFL standings is "remember: regression to the mean," in MLB, might it be, "stick w/ the horse that won last year"?
I think the length of the MLB season negates a lot of luck. Even though the better team wins more often in the NFL than MLB, the NFL 16 game season is so short that there is a lot of luck in team records. Regression helps hedge against the luck.
How can you say these things with a sample size of less than 1 season? The number of seasons seems alot more important than number of analysts in this study.
"My point is that as we approach football’s silly season of predictions, don’t believe anything you read or hear. Don’t put much stock in a favorable prediction for your team, and don’t panic at an unfavorable one."