tag:blogger.com,1999:blog-38600807.post3814381443512554616..comments2023-03-09T10:28:21.809-05:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): Pre-Season Predictions Are WorthlessUnknownnoreply@blogger.comBlogger11125tag:blogger.com,1999:blog-38600807.post-86431228032099796292009-11-27T02:02:21.990-05:002009-11-27T02:02:21.990-05:00Do you think it makes more sense to regress teams ...Do you think it makes more sense to regress teams that have established winning continuity less than teams that seem to have more volatile records?<br /><br />Lets say you and I are going to bet one another a dollar per game in which we're off on our predictions ie. if I predict the Panthers to win 11 games and they win 9, I owe you $2. We make our picks in separate rooms at the same time to avoid any betting strategy based upon what we know our opponent would do.<br /><br />What would be your strategy? Your data here seems to suggest that you'd choose mindless 8-8 predictions across the board since its error is the same as the "experts" (ha!), and you'd rather not spend the time it takes trying to come up with an algorithm to produce future wins if it'll be no better. But would you really bet like that?<br /><br />I have to believe that picking the Colts to win 9 games, everyone else to win 8, and the Raiders to win 7 would probably net me $2 against that system.Marverhttp://pigskintelligence.blogspot.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-70757427988242976532007-09-07T19:56:00.000-04:002007-09-07T19:56:00.000-04:00yeah -- sorry muddled projections wasn't very clea...yeah -- sorry muddled projections wasn't very clear. FO projects the probability of wins by bucket. Some of the teams are very clearly projected to be very good or bad -- Philly ~90% in the 9-10 plus 11+ buckets; KC ~70% in the 0-4 plus 5-6 buckets. Some team's projections are ~equally spread through all buckets, like Minn -- 10% in the 0-4 and 11+ and ~25% in the others. If you throw out these (the muddled ones), the projections look a bit better. This likely also explains why FO keeps winning the Slate competition -- its good at picking the division winner, but very mediocre at picking everyone.Unknownhttps://www.blogger.com/profile/06893515504632890107noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-31715465093655222862007-09-05T22:10:00.000-04:002007-09-05T22:10:00.000-04:00Thanks for running that analysis on the 8-win pred...Thanks for running that analysis on the 8-win predictions. I didn't really feel like it to be honest. I'm surprised how little value the FO predictions add. But I think this is why it's important to publish your error rates and accuracy metrics. If you're trumpeting what you got right, you need to be forthright about what you got wrong. It's not good business (and clearly FO has built a reputation for good preseason predictions if TMQ is using them), but it's good science.<BR/><BR/>I was curious about your take on the <A HREF="http://www.footballoutsiders.com/2007/09/05/ramblings/dvoa-rankings/5437" REL="nofollow"/>. These seem to be adding error on top of error (the preseason VOA projections and then adjusting for strength of schedule based on those projections). Honestly, I can't imagine Miami having the worst offense in the league. That's my own baggage, I know, but I honestly don't understand why Trent Green is projected to do so poorly.<BR/><BR/>And by muddled predictions, I imagine he means New Orleans.Derekhttps://www.blogger.com/profile/17941314072950152029noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-52824548444436763992007-09-05T18:29:00.000-04:002007-09-05T18:29:00.000-04:00Miles-I saw that post, but I don't understand what...Miles-I saw that post, but I don't understand what you mean by muddled predictions.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-82707913471854675812007-09-05T12:57:00.000-04:002007-09-05T12:57:00.000-04:00Here's a similar discussion on the FO site: http:/...Here's a similar discussion on the FO site: <BR/>http://footballoutsiders.com/discussion/viewtopic.php?t=146<BR/>I did find the FO predictions to be better (in 2006) when ignoring the teams that had muddled predictions, but only marginally better in 2005. Not a whole lot of data to use.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-57903490342756896462007-09-04T23:33:00.000-04:002007-09-04T23:33:00.000-04:00Derek, I agree MAE is only one way to look at it. ...Derek, I agree MAE is only one way to look at it. But I disagree that 17/32 within 2 games, and 10/32 within 1 game, is any good.<BR/><BR/>I ran the same comparison for the "mindless" 8-win prediction method (which I guess should be called the "null prediction"). <BR/><BR/>Over the past 2 years, it was within 2 games an average of 16.5 times, and within 1 game an average of 10 times.<BR/><BR/>For 2006 it was within 2 games 16 times and within 1 game 10 times. <BR/><BR/>By your standard, FO adds a single game of valid prognostication over a 256-game season. It took dozens of FO "game charters" and ultimately thousands of man-hours of analysis to come up with their predictions. That's not so good. In fact, some could call it embarrassing.<BR/><BR/>But I do agree with your point that it's really the division rankings that matter.<BR/><BR/>By the way, thanks for the great original post on this matter.<BR/><BR/>Also, I think everyone will like tomorrow's post. (Tarr-you're on to something.)Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-45426420797098216892007-09-04T23:03:00.000-04:002007-09-04T23:03:00.000-04:00I think there are other ways to look at it besides...I think there are other ways to look at it besides mean absolute error. By MAE, yeah, they're no good. Correlation coefficient is another way to look at it. <A HREF="http://footballpredictionnetwork.blogspot.com/2007/08/oversight-accuracy-of-pro-football.html" REL="nofollow">I looked at it another way</A> (as I know you've read) and found that 17 out of the 32 predictions were within 2 games of being correct, 10 of those being within a game. That's not bad actually.<BR/><BR/>And maybe it's not the exact win totals that matter as much as the projected division rankings. Or maybe you could see how often the win total range with the highest projected probability turns out to be the correct range.Derekhttps://www.blogger.com/profile/17941314072950152029noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-30708835362180151982007-09-04T15:32:00.000-04:002007-09-04T15:32:00.000-04:00Chip-By "last year's record" I mean a simple regre...Chip-<BR/><BR/>By "last year's record" I mean a simple regression based on last year's record, not just using a team's previous wins as the projection for next year. See the end note for the formula.<BR/><BR/>What this does is simply apply a regression to the mean to last year's wins. So a prediction for a 13-win team would be about 11 wins, and a prediction for a 2 win team would be about 5 wins.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-34516268879575323222007-09-04T15:26:00.000-04:002007-09-04T15:26:00.000-04:00FO's projections are not great, but not worthless ...FO's projections are not great, but not worthless either.<BR/><BR/>I looked at the Pearson rank correlation for the FO's and Vegas' projections as well as last year's record: <BR/><BR/> 2006 2005<BR/>FO 43% 33%<BR/>Vegas 45% 20%<BR/>Last Year's 30% 20%Chiphttps://www.blogger.com/profile/15848013553962095565noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-60389967342919340292007-09-04T14:38:00.000-04:002007-09-04T14:38:00.000-04:00The only pattern I recognize is that expert picks ...The only pattern I recognize is that expert picks tend to go out on a limb. They need to validate their existence, so if they pick teams to win within a range of 6-10 games, no one would bother paying attention. But the smartest prediction strategy is to go as short out on a limb as possible (like the 8-win strategy).<BR/><BR/>I'd put money on teams that Vegas has as outliers: 5 or less wins and 10 or more wins. I'd bet the under on the good teams and the over on the bad teams. But for now that's a hunch.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-17140839204905668682007-09-04T12:00:00.000-04:002007-09-04T12:00:00.000-04:00Is there any discernable tendency if we look at th...Is there any discernable tendency if we look at the predictions where Vegas/FO/8-8/regression to the mean disagree strongly with one another?<BR/><BR/>Let me give an example. I have no intention of wagering on team wins. But if I were to place such a wager, I would look for teams where:<BR/>- The Vegas over/under and the FO prediction differed by more than 2 games.<BR/>- The Vegas over/under was far off from 8-8, and the FO prediction was much closer.<BR/><BR/>Is this strategy any better, statistically, than taking the over on teams with o/u<6, or the under on teams with o/u>10?Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.com