tag:blogger.com,1999:blog-38600807.post7434876616084393012..comments2023-11-05T04:16:44.937-05:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): How Accurate Is The Prediction Model?Unknownnoreply@blogger.comBlogger15125tag:blogger.com,1999:blog-38600807.post-25303963268377089602011-11-11T16:33:35.761-05:002011-11-11T16:33:35.761-05:00David, dumb question here, but do your predictions...David, dumb question here, but do your predictions take into consideration the line, or is your analysis straight up, team vs team?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-5090198839199867712010-10-14T15:16:33.302-04:002010-10-14T15:16:33.302-04:00Do you have any data showing whether the second ha...Do you have any data showing whether the second half of the year is more accurate than the first couple weeks when there isn't as much data or does the model have enough data to start being just as accurate in week 4 as it does in week 15? Obviously last weeks results were a good indication that the early week results are pretty good.Buzznoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-16367250929826761772010-10-14T00:32:10.491-04:002010-10-14T00:32:10.491-04:00Dave, if we're talking about overall predictio...Dave, if we're talking about overall prediction accuracy, a good measure would be root mean square deviation. So, for each game, take the predicted percentage chance given to the home team, subtract the result (100% or 0%, or 50% for a tie), and square the result. Average the squared deviations of all predicted games, and square root the average.<br /><br />Benjamin Morris, I'm not saying I would have predicted that, but I can make a reasonable-sounding justification for why 3rd>1st>2nd>4th quarter of the season makes a lot of sense for predicting NBA results. I wouldn't be surprised if the order came out the same for the NFL, for roughly the same reasons.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-28822039980820235462010-10-13T14:10:16.879-04:002010-10-13T14:10:16.879-04:00In my research, I've found that not only is im...In my research, I've found that not only is immediately previous performance less reliable than overall performance, but it's not necessarily even more reliable than less recent performance. E.g., in Basketball, the 3rd, 1st, and 2nd quarters of the season are actually more predictive of playoff success (in that order) than the 4th.Benjamin Morrishttp://www.skepticalsports.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-8968265455814640912010-10-13T09:40:53.018-04:002010-10-13T09:40:53.018-04:00Jim-Yes, in fact in late 2007 and all of 2008, the...Jim-Yes, in fact in late 2007 and all of 2008, the model overweighted the last 4 games (by about double). It turned out to be no more predictive than using the entire season evenly. My rule is, all things being equal, go with the simpler more elegant method.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-53564255427955935772010-10-13T09:06:45.210-04:002010-10-13T09:06:45.210-04:00Brian - have you ever looked at using the efficien...Brian - have you ever looked at using the efficiency stats of the past 4 games for the prediction model as opposed to the whole season? I was reading "Take your eye off the ball" and they mentioned in there that coaches game plan off the past 4 games of their opponents. Either they are operating inefficiently there by not taking all the data available (which should be relatively easy these days) or they understand that the past 4 games of a team are more accurate than what they did several months before. I guess the best way to evaluate would be to see which has a better correlation to game stats, efficiency stats over the whole season or from the past 4 games.Jim Fisherhttps://www.blogger.com/profile/03232518849821522070noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-39318153398037104322010-10-12T23:37:01.836-04:002010-10-12T23:37:01.836-04:00I blogged briefly about this here:
http://skeptica...I blogged briefly about this here:<br />http://skepticalsports.com/?p=839<br /><br />Unrelated, Dave: Your point is valid and important. I'm working on a calibration method for these types of models that rewards accurately making bold predictions. It's more difficult when the outcomes are binary (as with wins rather than winning percentage per season), but it's done all the time in finance and other quantitative fields.Benjamin Morrishttp://www.skepticalsports.comnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-89140063581297089122010-10-12T01:24:21.375-04:002010-10-12T01:24:21.375-04:00"I actually want the model to be 'wrong&#..."I actually want the model to be 'wrong' 40% of the time."<br /><br />I get this point, but while this displays the model's accuracy, it doesn't necessarily say much about its precision. For instance, I could give a model that says "Home team always has a 57% chance of winning" and it would backtest extremely well, and likely do very well in the future. A better way to compare models would be to judge its accuracy and somehow account for how often and how far it strays from the safety of 57%. In an extreme (although impossible) case, the best model would be one that always predicts 100% or 0% for the home team and is always right.<br /><br />Does that make sense? Is there someway to measure that aspect of your model? Might not be too meaningful in a vacuum, but would be a good way to compare against other models.Davenoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-90698342454404859382010-10-11T23:29:40.272-04:002010-10-11T23:29:40.272-04:00Not quite. The probabilities are for the home team...Not quite. The probabilities are for the <i>home</i> team. Not complimentary.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-75529570431869235662010-10-11T22:55:32.608-04:002010-10-11T22:55:32.608-04:00Reduce confusion and plot either the greater than ...Reduce confusion and plot either the greater than 50% or the less than 50% since they must be complementary. However you represent it, linear regression would give the appropriate corrections which when applied to existing data gives the actual accuracy of the methodology.Unknownhttps://www.blogger.com/profile/11602198685138920900noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-43617738133627744432010-10-11T20:33:56.865-04:002010-10-11T20:33:56.865-04:00Do you have an archive for this prediction model?Do you have an archive for this prediction model?Chuck Winklernoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-32938410450386469922010-10-11T15:02:54.498-04:002010-10-11T15:02:54.498-04:00Just from eyeballing that, I'd say it appears ...Just from eyeballing that, I'd say it appears as though the system is a bit overconfident. A linear fit of the actual win rates would have a lower slope than 1:1, which means your predictions would be more on-target if you nudged all the predictions a bit towards 50%.<br /><br />It also seems like you're very very slightly undervaluing home field, but that could just be my eyes playing tricks on me or an artifact of the small sample size.Tarrhttps://www.blogger.com/profile/14368810359650066790noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-39218162676688275872010-10-11T08:00:43.737-04:002010-10-11T08:00:43.737-04:00You should just submit your predictions/ratings to...You should just submit your predictions/ratings to Todd Beck's Prediction Tracker at http://http://www.thepredictiontracker.com/<br /><br />That way you can have you have your model tracked independently against dozens of other prediction models.Jim Anoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-67417016968254661832010-10-11T05:43:59.211-04:002010-10-11T05:43:59.211-04:00A lot of people are curious not just about the gam...A lot of people are curious not just about the games that win but the games in which your model's opinion is vastly different than the conventional wisdom. The Vegas betting lines can be converted into percentage chances of winning the game straight up. I think a lot of us are curious about how often the model works where you and the oddsmakers has really different opinions. <br /><br />For example, you gave Cleveland a 72% chance of winning where Vegas said about 40%. You had Arizona at 52% where Vegas had them around 25%. <br /><br />These cases are far more interesting than the ones like Oakland where both said an upset was unlikely and one occured.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-27367539821082158872010-10-10T21:37:59.245-04:002010-10-10T21:37:59.245-04:00Was this written/conceived before your latest NY T...Was this written/conceived before your latest NY Times article because this is similar to what you mentioned there.<br /><br />Are you going to run the numbers for more seasons to see if it is consistent and will you change your algorithm as a result?<br /><br />I'm interested in how the predictions shake out for the rest of this seasson as there have been some crazy wins and losses so far.Gileshttps://www.blogger.com/profile/17448803189162178366noreply@blogger.com