"Analytics" has unfortunately become a trendy buzzword in sports. I've found that many people who are only vaguely familiar with analytics, including some team executives, media members, and fans, have the wrong idea about what analytics is. Some think it's a panacea that can optimize the solution to any problem. Some think it's just statistical trivia or scientific minutia, like ESPN's Sports Science series (Dwight Howard's arm-span is as big as a 2-car garage!). Others think it's just Moneyball, a one-time talent arbitrage applicable to only one sport. So I thought I'd put my own thoughts down on what analytics is, at least as it applies to football.
I'm not the gatekeeper on what qualifies as analytics, and I'm not going to say what counts and what doesn't. But I think analytics comprises all or parts of four general processes:
1. Asking meaningful questions
2. Getting relevant data
3. Analyzing the data quantitatively
4. Presenting the results
Sports analytics is really no different than any other statistical field of inquiry. These 4 basic processes are basically just the classic scientific method. Like many other fields, we can't really create lab experiments. If we want to know if calling a timeout to prevent a 3rd-quarter delay of game is worth it, we can't replay the same game 1,000 times with the timeout and 1,000 without the timeout, and compare which choice leads more often to winning. In sports, the 'natural' experiments have already been done for us, each one as a unique trial.
The bottom line of analytics is that, if done well, it can provide concrete insight to make better decisions. Just like the Scientific Revolution we learned about in grade school, sports analytics is replacing what used to be often-mistaken intuitive judgments and conventional dogma with rigorous analysis.
Analytics is different than traditional statistical analysis in two ways. First, the digital age has made large amounts of data widely available. "Big Data" can fundamentally alter the scientific method. With a large enough set of data, a traditional hypothesis isn't always required, at least for many purposes. It's also enabled the spread of novel ideas and methods far beyond what a book or magazine article could do. Second, computational technology has progressed to the point where relatively advanced machine learning algorithms are widely available and are nearly costless to apply. (Just compare Virgil Carter feeding punch-cards into a university time-share mainframe decades ago to a personal laptop that can compute the results of thousands of model specifications in seconds.)
I see three distinct branches of sports analytics, each with its own value and each overlapping the others.
Sports Analytics Venn Diagram
These sets aren't intended to be to scale. It's not even close, as the Things Interesting to Fans domain spans topics like fantasy football and gambling. Most readers probably identify this area with the kinds of things that popular sites like Football Outsiders or Football Perspective focus on and do very well: team rankings, "best of all time" comparisons, historical statistics, forecasts, and in-depth conventional analysis with a statistical bent. This area is often the entry-point for fans interested in better analysis than what they traditionally get from the media.
The Things Interesting to Science domain includes research on human performance, competition, cooperation, and decision-making under uncertainty. Many of the academic papers on sports are ostensibly aimed at learning something broader about a non-sports scientific topic. Sports can offer a natural laboratory setting with a reasonably controlled environment that researchers can't find elsewhere. A football game is bounded by the clock and end zones, with uniform rules and clear zero-sum objectives. It's also measured and recorded in all kinds of ways. You don't find that in many other natural human settings.
The Things Helpful to Teams domain is probably the hardest to crack. This is where I've been focused in recent years, and it's what I enjoy despite its challenges. This area is almost all overlap--there is probably very little that is helpful to teams that fans aren't also interested in, at least for the fans who are readers at ANS. This domain is all about helping teams make better decisions, on the field, on the sideline, or in personnel meetings.
I'd like to think that ANS is usually right in the middle of the diagram where all three sectors overlap. (Of course I do, I drew the diagram!) Sure, we do fun stuff that is of little use to teams like rankings, best-of comparisons, and game predictions. But I like to focus on the overlap. For example, the Win Probability model can help us decide if Kurt Warner should go into the Hall of Fame (something of interest only to fans), but it can also help coaches make better in-game decisions (something helpful to teams). It can also help tell us whether people are intuitively good at playing minimax games (something interesting to science). It all goes back to step one above: asking meaningful questions.
There's a 4th area of sports analytics I've left out, but it's probably much larger than everything else (except gambling): sports business analytics. This area is about marketing, ticket prices, tv ratings, branding, fan experience, when to have bobble-head night, and stuff like that. Although analytics has thoroughly penetrated this area, there's not much about it that's unique to sports. In the end, sports is just part of the entertainment industry, and at one level it's not terribly different than putting on a series of live concerts or Broadway plays. I'm primarily interested in the "sports analytics" that has to do with the competition on the field (including roster construction and associated financial considerations) rather than in general business analytics, despite the overlap in skills and tools required.
I was inspired to create the sports analytics diagram by one that author and analyst Drew Conway made to describe the skills needed in data science. (By the way, I recommend Drew's book Machine Learning for Hackers--but be warned: learn R first.) In any field including sports, data science requires three types of skills: hacking skills (deep computer skills plus an instinct toward tinkering), math and pure statistical skills, and some amount of subject matter expertise. (I particularly like the Danger Zone warning...There are a couple pieces of football research that immediately come to mind. However, I would argue the "machine learning" intersection is another danger zone in football analytics. But that's a whole different discussion.)
When I began crunching football stats (at bbnflstats for those who remember) I knew enough regression to be safely out of the danger zone and that's about it. I was probably in the 'Traditional Research' intersection. Over the past 7 years, I've learned at least bits and in some cases a great deal of the following: probability, statistics, optimization, modeling, several machine learning techniques, behavioral economics, utility theory, decision theory, game theory, network theory, heuristics, R, PHP, CSS, Javascript, Java, C#, Python, mySQL, AJAX, Visual Basic, XML, Flash, visualization skills, and of course a great deal about the sport of football itself.
What makes "Analytics" so interesting is that spans so many interesting topics and useful disciplines that it never gets boring. But that's also why it's hard to define.
Interesting article, Brian. I certainly have no problem with the idea that Football Outsiders concentrates on "Things Interesting to Fans." As the founder, I've always considered myself a writer who tells stories with analytics, rather than an analyst who writes. And of course, both our humor columns and our scouting columns are there to be interesting to fans without really representing analytics.
I've always said that while NFL teams are behind on a lot of things (fourth downs, of course, most of all), football teams needed a revolution a lot less than baseball teams did. The salary cap always forced them to think more about issues of economic efficiency, and constantly changing rules forced them to live without a "book" of strategies from 100 years ago. But the football media was horrible a decade ago. On print and on the Internet, at least, it is better now. On television, not really. My goal has always been better coverage, not better team management.
I'll just send the teams over here when they need to know when to let the other team score late in the game. :)
Really nice summary, Brian.
That's a good summary, Brian. It instantly brought me back to an e-mail I received in July 2004, after I asked Drinen for some tips on writing fantasy articles. Here was his advice:
--------------------------------------------------------
In terms of conveying high-tech stuff, I always like to walk the reader through my thought process. It usually goes something like this:
1. Here is a question, and the fantasy implications of the question
2a. Here is a way to use data to answer that question
2b. Here are the problems with what I've come up with in 2a.
3. Here is the data. (if the data set is large, this can come after the conclusion, as sort of an appendix.)
4. Here is a conclusion.
--------------------------------------------------------
I think of analytics the way a writer would think of analytics, which is the way Doug thought of how to write an article. Your 4-step approach just about perfectly mirrors what Doug said to me ten years ago. I'll add that there are two things Doug wrote that deserve special emphasis, even if they're a sub-category of your generic steps:
-- presenting the problems with your data. This is probably the biggest thing we, as drivers of the analytics movement, fail to do well. When making an argument, people don't want to focus on the potential errors or reasons the analysis doesn't apply or the data do not fit.
I certainly try to be cognizant of this and point out the problems with my assumptions to my readers, but this is certainly one of the things that can slip when you're rushing to get an article done or trying to publish an article every day. I'll leave it to my readers to tell me if I'm doing a good job here, or more importantly, to call me out on it when I don't.
-- "here is the data."
This is another tricky one. FO has their own proprietary data, which makes it hard for readers to independently verify their work. PFF has subjective grading based on hundreds of thousands of plays that is simply impossible for anyone to verify. Personally, I have lots of data that are not publicly available, but I'm not in the position to just publish the data either, since it is not my own. That makes some (a lot?) of my work unverifiable, either. ANS is sort of in the same boat. I think you do a good job publishing what you can, but it's not possible to verify a lot of your studies. It's a tough balance for all of us in this industry.
Anyway, a very interesting topic, Brian. More to your main point, I agree that most people (particularly the detractors) don't have a clue what "analytics" really means. To me, analytics mostly means "thinking about things objectively and rationally" while applying the four- or five-prong tests we've discussed.
Sorry that this is off topic, but did you happen to see Gelman's posts the last couple of days regarding modeling scores rather than wins?
http://andrewgelman.com/2014/02/26/econometrics-political-science-epidemiology-etc-dont-model-probability-discrete-outcome-model-underlying-continuous-variable/
The basic idea is that you throw out information when you reduce it to a binary outcome. As someone who has previous defended modeling win probability because it's what ultimately matters, I was wondering if you had thoughts on this.
Here are my immediate thoughts, but to be honest Gelman has a good point that I need to think about some more.
Ultimately, WP is the risk-neutral objective function of any sport. It's pure linear utility, so we want to get there the best way (with least error/bias) as possible. The only question is whether we get there directly or indirectly.
I think there are advantages to both approaches. I advocate both and I use both. However, the score-approach might work well for other sports where it fails for football for several reasons. For starters scoring is not a continuous variable in football like it is in baseball, hockey, soccer, and (effectively) for basketball. Scoring comes in chunks, so it's distribution forward from any given game-state is not easily modeled without making some very costly assumptions (pace, risk/reward, clock management for example).
I think Gelman is right when it comes to things like elections, which he studies, but sports can be different, especially football. Football offers a rich spectrum of risk-reward choices for competitors that elections (or baseball) doesn't offer. Romney can't go for it on 4th down or change his pace or try an onside kick. Obama can't kneel out the clock. Also, to a poli sci researcher a score is a poll, which has a lot of uncertainty. The election simply resolves that uncertainty. Sports scores are known exactly.
Getting to WP via score distributions is useful for some purposes, but it requires far too many assumptions about coaching behavior/risk-reward/etc to get there accurately, especially in the end game.
Really interesting and thoughtful post that gives me a lot to think about. Thanks. The comments there are great too. I always read Gelman but hadn't seen it yet.
Yeah, I agree about the odd nature of NFL scoring, mostly coming in 3s and 7s. And also that scoring as much as possible isn't the objective. I actually tried to raise that last point with regard to basketball in a question on a related post, but he did not answer:
http://andrewgelman.com/2014/02/25/basketball-stats-dont-model-probability-win-model-expected-score-differential/
I suppose his answer would be something like: any attempt to use clock instead of scoring more would already be captured in the data. But that's certainly a difference between sports with clocks and say baseball where there's really no reason to put on the brakes ever (although you do occasionally skip an entire half inning, which I guess is weird).