I came across this article from a website called Trending Buffalo just yesterday. It's from September, but unfortunately it's one of the top 10 results if you google football analytics. Although the author is misinformed or uninformed or both, I sense that his feelings are shared by a large number of fans and traditional analysts in the media. So I thought I'd respond point by point.
From 'Trending Buffalo' Sep 13, 2013
http://www.trendingbuffalo.com/sports/buffalo-bills/5-reasons-football-analytics-arent-as-great-as-you-think-they-are/
1. YOU DON’T KNOW WHAT YOU DON’T KNOW
Precise breakdowns of every player on every play of every game can be found on countless websites [countless?] but “Player X failed to execute his route” and “Player Y is responsible for the B gap” ring pretty hollow when you don’t know what the players were supposed to do. [The author doesn't understand what analytics is. It's not amateur scouting.] Remember when Doug Flutie discredited analysts because “You watch the game. We watch the film”? He should’ve added “and we know the play calls.” Without that crucial information, your analysis is a really, really in-depth GUESS. [Football analytics is a lot of different things, combining tools from several disciplines, but it's not about guessing assignments on plays. I suppose the author doesn't care for Pro Football Focus, which isn't strictly 'analytics' on its own. But I'd submit that even though PFF doesn't know 100% of the assignments, most can be easily inferred. It's far from perfect, but it's a big step forward for public player evaluation and a giant leap beyond traditional punditry.]
2. WHAT HAPPENED 45 MINUTES AGO IS PROBABLY MORE RELEVANT THAN WHAT HAPPENED 45 YEARS AGO [What's the chance the author will get a green light at the next intersection? Is he 100% certain he will get the same result he got driving through it 45 minutes ago? Or is the best estimate based on a long-term average of recent history?]
You want to tell me I should’ve gone for it on 4th and 2 because of the success rate of the average offense versus the average defense on the average day since the merger? [I've never seen any analysis of 4th downs using data since the merger. The earliest data I've used is from 2000. However imperfect league baselines may be, they are still a massive improvement over a coach's intuition, and at the very least should calibrate a coach's risk tolerance.] Well, what if my offense is below average and my opponent’s defense is above average and the perennial Pro Bowl defensive tackle is eating my rookie interior linemen alive and my best available RB is nursing an ankle sprain? What do your numbers tell you now? [In that case, analytics can adjust the numbers for team strength, and further it can provide a coach a 'break-even' success rate, above which it makes sense to go for it.] Fact is– you’re not telling me what’s going to happen to me. You’re telling me what already happened to others. This is useful information and I’ll treat it as such… but that’s where it ends. [If someone told the author that 20% of the people who go swimming at that beach are attacked by sharks? Would he discount that information and swim there because it's something that "already happened to others?"]
3. NOW NERDS CAN PLAY! [Nice smear.]
Football is a big, tough manly sport and analytics provide[sic] a way for all of us to get involved without getting hit. It’s appealing. We value our ability to think our way through the game and, after the fact, it makes a lot more sense to say “I wouldn’t have done what that coach did” than “I wouldn’t have thrown that interception” or “I wouldn’t have fallen for that juke move.” [The author tries to define football analytics here as outcome bias with numbers. Knock that straw-man down with all your might!] We can imagine ourselves in the role of coach. We all have Madden. We all have Excel. Plugging flawed data (see #1) into a spreadsheet in an attempt to accurately predict results is a fool’s errand. [All analysis using flawed data would be a waste of time. What about when the data isn't so flawed?]
4. IT’S (MOSTLY) PSEUDO-SCIENCE
I’m 100% in support of professional teams utilizing all of the data at their disposal in an effort to make the best decisions possible [Ok, great. So we are in agreement] but it all breaks down when we get to the amateur/armchair level. [With some exceptions, the amateur/armchair level is two decades ahead of the pro teams in terms of successful applications of analytics.]
Pseudoscience is a claim, belief, or practice which is presented as scientific, but does not adhere to a valid scientific method, lacks supporting evidence or plausibility, cannot be reliably tested, or otherwise lacks scientific status. Pseudoscience is often characterized by the use of vague, contradictory, exaggerated or unprovable claims, an over-reliance on confirmation rather than rigorous attempts at refutation, a lack of openness to evaluation by other experts, and a general absence of systematic processes to rationally develop theories. A field, practice, or body of knowledge can reasonably be called pseudoscientific when it is presented as consistent with the norms of scientific research, but it demonstrably fails to meet these norms.[Football analytics is precisely the opposite of what the author quotes above as pseudoscience. Specifically, analytics seeks to take sports analysis out of the realm of the intuitive and the post-hoc narrative, and subject it to rigorous scientific methods. It's objective, precise, consistent, evidence-based, makes falsifiable predictions, and is open to evaluation. There are some kooks out there who claim to be doing analytics, but that's a fact of life in any field of inquiry.]
The lack of reliable data (primarily because of the inability to use a control group to isolate specific events within the context of an 11 on 11 game) severely injures the credibility of any conclusions derived from said information [Huh?]. So, the average NFL play nets 4.8 yards? How many yards is the average team attempting to gain on each play? 2? 4.8? 80? Without that knowledge, we have nothing. [Analytics is not about predicting the outcome of the very next play or mind reading. Where did the author get that impression?]
5. IT’S GETTING WORSE [It's gaining traction because it's useful. And it apparently worries the author because he doesn't understand it, as his article demonstrates.]
Looking at data before making decisions is a positive development and I hope every team I support keeps up with the times. [The author should just stop right there. That's all analytics claims to be. There are statistical and analytical techniques that make sense of that raw data, but those same techniques are applied successfully in all fields of industry, government, and science.] But the further we go down this road in popular/public/amateur conversation, the more bad information creeps in. Suddenly, the word “analytics” starts to sound a lot like “run and stop the run” and “defense wins championships.” Research all you’d like but we’re getting into “He made the wrong decision and I can prove it” territory (even though you can’t) and it’s growing tiresome. The day that “Analytics Guy” [I wonder who he's referring to?] realizes his information is helpful but not definitive, everybody wins. [Well, somebody is winning this debate, and it's not the author of this article.]
And besides, if you don't care for the kind of analysis here at ANS, we helpfully provide an alternative.
Great takedown, Brian! Let me guess...the author of that article doesn't accept or respond to rebuttals? That's usually how this thing works...hit and run and hope no one hears the other side of the argument.
That author must have hit his word limit, otherwise I'm sure he would've gone to the old, "What do you know, I'll bet you never even PLAYED football." And don't forget the, "What next, computers calling the plays?" argument. Other than those two, he pretty much hits for the cycle on ignorant arguments.
To me, it sounds as though the Trending Buffalo author's problem is not with football analytics but with statistics in general. Though he tries to bring his arguments back to football, his belief that we cannot use past data to analyze future outcomes seems to contradict the purpose of statistics and science in general.
This sounds about as informed, well-reasoned, and wishfully composed as Karl Rove's rants against Nate Silver on election night in 2012! Maybe this Buffalo needs Megyn Kelly to walk him back to the decision desk...once in a while the "real world" tends to have exactly the same bias that traditional pundits think is lingering in the data, a phenomenon that is decidedly inconvenient for the "experts."
Headline for "Week 15 Bills Podcast" by Brad Riter:
The playoffs are starting to seem unlikely.
He probably didn't need analytics to tell him that either, considering they were 4-9 at the time.
Is the point of this article that we shouldn't use statistical arguments to second guess coaches? Are we allowed to second guess coaches by any other means, such as gut feel or heuristics?
How do you explain all of the countless,obvious referee mistakes?Go to Thefixisin.net,click on "Updated 1-2-14 news of note."And please explain all of that to me.
Well, the guy isn't literate, but he might be groping his way toward cogent points. If you translate points #1 and #2 into normal language, he may be trying to say:
1. Current analytics do not capture the correct degrees of freedom. Since the primary reason for a play being successful or not is whether the individual players execute their roles in the called play, a broad analysis of the performance of a player over all kinds of plays against all kinds of opposition can never have high predictive value.
2. The specific composition of the team on the field and knowledge of available play-calling are more important than generic down/distance information. If 80% of all 4th and 2's are being converted by teams that have strong O-lines and make short runs down the middle, then if my team lacks strength in that area, the generic information about 4th and 2 success is not useful to me.
Is it certain that these two criticisms are inaccurate? We should be able to quantify the variance in our predictions due to (at least) the total of the unknown degrees of freedom. There may well be important factors that are not being captured.
Agree with everything said...
Just want to point out that the following is the heart of the anti-analytic, rely on the coach's gut camp's argument...
"Well, what if my offense is below average and my opponent’s defense is above average and the perennial Pro Bowl defensive tackle is eating my rookie interior linemen alive and my best available RB is nursing an ankle sprain? What do your numbers tell you now? "
This is the heart of their argument and it sounds much more logical than it actually is. I feel like our side loses people on this argument more than any other. We really need to spend more time fighting it.
Most people on here already know the answer but sometimes we don't give it as often as we should. The truth is that you do need to adjust for the team's strengths but you need to put those adjustments in for the entire rest of the game and not just the next play. Sometimes we forget, but the argument to punt is really an argument that you are better off if you give the ball back and attempt to get it again. You need to make a stop on defense and then get the ball and do something with it once you have it again. If your offense is so bad that they have no hope on one play, what makes you think they can do anything later in the game.
When you run through the math, you see that team strength adjustments end up as a wash and that you end up with the same basic conclusion for fourth down the vast majority of the time. It actually makes sense when you think about it, but somehow most people only want to apply the adjustments in a vacuum to the upcoming play.
Jeez, it's a satire/comedy site. You spent a lot of time and you should probably end up on literally unbelievable. Trending Buffalo is a satirical blog, and you couldn't have taken 5 minutes to read maybe 2 other posts to realize that? Wow, just... wow...
You took that WAY too seriously Maybe check out the rest of the website first before posting something like this. Just about everything they write is tongue-in-cheek. Maybe you just need to be from Buffalo to understand.
You think the author wouldn't respond to a rebuttal. Maybe not, but he sure is happy you wasted your time on him and posted as facebook that they'd won their first award of 2014.
He's also a Bills fan (as am I), and you can shove your analytics. I don't care if there's an above 50% chance a team converts a 4th and short, because if the Bills are playing the Patriots, it ain't gonna happen.
Although I could do without the Trending Buffalo author's overwhelming negativity towards analytics in general (it reads almost like a religious fundamentalist decrying science), I'm disappointed that the author of this rebuttal basically ignores the first author's main point, which I think is an important one:
Analytics is often used incorrectly (or at least could be used more correctly) yet presented as infallible truth.
For context, the Trending Buffalo author is implicitly challenging a few outspoken radio personalities from Buffalo's local sports talk radio station who, often condescendingly, adhere to certain analytics-inspired opinions which, on the surface, seem disconnected with reality. “The only play teams should EVER run on 3rd/4th-and-short is the QB sneak," or "teams should always go for two," or "teams should never punt," to mention a few.
The obvious problem with this kind of thinking is that it presents analytics as infallible, when in reality analytics – ESPECIALLY at such a young age – should be treated as anything but. If analytics is to assume its deserved relevance in sports analysis, it needs to be characterized by thoughtful debate, not some contrived notion of pro-analytics versus anti-analytics.
Basically, the conversation should be about improving the application of analytics, not arguing for or against it in general. Credible scientific theories are improved upon or even flatly disproven all the time, and analytics people need to embrace this ethos of humility and skepticism to give their movement a broader appeal.
I'd say points #1 and #2 are bang on. The fact is that years long /league wide averages do not apply specifically to any one team. Local effects, weather, injuries, etc have large effects.
Point #3 is irrelevant. Point 4, it is not pseudoscience at all, it is an accurate description of past events - however the interpretation of the analysis can enter into "pseudoscience" when they are used to predict the future, and to conclude that a specific decision was "wrong" based on small changes in estimates of WP.
Oh, actually having thought about it for a second, it is indeed pseudoscience.
Any of these predictions and conclusions are non-falsifiable. No one can ever prove that the play call to go for a pass on 3rd and 1 was the wrong call, or the right call.
Science must be falsifiable.
To the guy above who thinks the original article was satire...Yes I took 2 min to read other posts, and you're completely wrong. It's not intended as self-satire. Mindless criticism and satire are not the same thing.
And to the clown immediately above...It doesn't sound like you understand what falsifiable means, or what analytics does. But at least we agree science must be falsifiable. You're a moron--falsify that.
Hey Brian - the Patriots outperformed their expected wins again...is Belichick still cheating?
Knd of like the Athletics in baseball, who seem to be the one team that consistently outperforms their nerd criteria.Very good chance that they and the Pats are well ahead of the curve n terms of player evaluation analytic techniques.
To me they sound like they don't believe in statistical analysis. Replace sports analytics with business analytics, would they have the same attitude?
"You're a moron--falsify that"
wow.
" I feel like our side loses people on this argument more than any other. We really need to spend more time fighting it."
my god, can you people hear yourselves? I am a huge fan of these analytic sites, but I am a bit disturbed when it becomes filled with religious zealots waging war against the "them".
PS i think it is an extremely valid point. The fact is, some teams are more likely to make 4th and 4 than other teams.
Lighten up!
Bill-That's a fair barb. I'll take that one. But I'll say this: We can debate its effects, but we do now know he was cheating through '06. That's not up for debate.
As a Buffalonian, I'm familiar with Brad Riter. He is a pretty legit sport media personality around the town and has been on and off local sports talk radio. I always thought he was fairly level headed and while not always agreeing with his points, I found them respectable. His piece wasn't statire, and I found it fairly disappointing. It was rightfully torn apart.
"Statire" <--was that intentional? Very funny. I think you just invented a whole new genre of sports journalism.
http://houston.cbslocal.com/2014/01/03/chris-kluwe-the-nfl-would-rather-have-felons-and-racists-and-abusers/
That comment thread makes me want to put my eyes out with a wooden spoon.
I'm still not giving the top QB's the ball on their own 45.Enter:punter.
Brian,
Not trolling; love the site. However, at first glance there does seem to be some possible inconsistency here.
(a) Analytics makes falsifiable predictions.
and
(b) "There are statistical and analytical techniques that make sense of that raw data, but those same techniques are applied successfully in all fields of industry, government, and science."
I agree that the data is being analyzed in the ways that you put forward. I agree that analytics isn't pseudo-science. But that doesn't mean that it is science (false dilemma alert). I don't have a considered view on the necessary and sufficient conditions to be a science and I'm not sure there are any that are illuminating; Popper went with falsifiable but some say string theory makes claims that aren't falsifiable. I would like to hear more on the falsifiable stuff. It's not as if Dr. Analytic says my hypothesis predicts that x will occur, x didn't, thus my hypothesis is incorrect (I know this is an over simplified version of what actually occurs in science), but instead Dr. A says that given what has happened before, it would be a good strategy to do x as opposed to y. If x doesn't work out, has Dr. A's claim been falsified? As I said above, I'm all for analytics, just not sure it's Science.
Analytics doesn't say things like "in this instance going for it would have been better than punting". It says that "given 1000 instances just like this one, most of the time going for it is a better idea". (or on average going for it yields more points, etc).
As for the argument that the quality of your offense or defense doesn't matter much, this is definitely not true.
Real world example: Chicago 2006 Defense vs Offense. For this team, Punting was not a means to 'give up', as it is with a team that has a good offense and a poor defense, but a way to force the other team to face more plays against a defense that was more productive producing points than your offense, or at least cause more kick returns where they also were more productive than the offense.
The decision to punt or not does need to factor in a lot of things, including your offense versus their defense, their offense versus yours, your special teams and theirs, and the situation in the game. The analytics show time and time again that coaches are too conservative and losing games on _average_ and this is scientific and can be falsified.
On any specific play, it is complicated and you can't scientifically say anything about one specific play, but on average it really is not complicated and is scientific.
This article sounds like a case for...Fallacy Man!
http://existentialcomics.com/comic/9
On ESPN today I saw the experts proclaim...
"In his entire long career, Peyton Manning has *never* lost twice in one season at home to the same visiting road team!"
Brilliant. Nice work Brian, as always.