Sam Waters is the Managing Editor of the Harvard Sports Analysis Collective. He is a senior economics major with a minor in psychology. Sam has spent the past eight months as an analytics intern for an NFL team. When he is not busy sounding cryptic, he is daydreaming about how awesome geospatial NFL data would be. He used to be a Jets fan, but everyone has their limits.
I see two legitimate rationales for immediately benching a running back who fumbles, one long-term and one short-term. In the long-term, a coach might argue, benching a fumbler will teach that player a lesson, inspiring him to improve his ball security for the rest of his career and help the team in future seasons. Meanwhile, in the short-term, the same coach might say, a running back who just fumbled is more likely to fumble on his next opportunity, and needs to sit on the bench so he can’t hurt the team right now. The long-term issue is harder to test, so here we’ll focus on the short-term concerns. If we want to test this reasoning, we need to find out if a running back with past fumbling problems is actually more likely to fumble in the future.
First, we can look at the consistency of fumble rates across seasons to see whether fumbling looks more like a repeatable skill or random noise. The autocorrelation for fumble rates, which measures the association between a player’s fumble rate in one season and his fumble rate in the next season, should give us a sense of this. Below we can see the relationship between running back fumble rates in “Year N” and “Year N+1” over the last ten seasons:
The autocorrelation of fumble rates for the running backs who had at least 100 touches in consecutive seasons over the last decade is 0.19. This indicates a weak, positive association between year-to-year fumble rates, so guys who fumble frequently one year somewhat tend to fumble frequently the next year. We can tweak our analysis by changing the touches cutoff or using more than one past season to predict Year N+1 fumble rate, but doing so yields similar results. It’s important to get a sense of fumble rate consistency on a seasonal basis, but this initial look at the data doesn’t give us a full picture of the short-term rationale for benching fumblers or its validity.
The short-term rationale posits that a running back who just fumbled is more likely to fumble again soon. Given two players who were similar in every respect up until the moment that one of them fumbled and one of them didn’t, the short-term rationale coach would opt for the one that didn’t fumble. But is there really any difference between these two otherwise identical players?
To answer this question, I used the Rubin Causal Framework to compare single-game fumble rates between players who fumbled the week before and players who didn’t fumble the week before, but had similar histories leading up to the game in question. (If you’re interested, you can read more about the Rubin Causal Framework here and at other Google-accessible places.) I measured player similarity using a player’s likelihood of fumbling in a particular game given their age, games started, touches, rushing yards, and fumble rate both over the previous three years and in-season leading up to the fumble in question. After controlling for these factors, I was able to find a point estimate and variance estimate for the effect of fumbling last week on fumble rate this week.
This method estimates that the effect of fumbling last week on this week’s fumble rate is a decrease of 0.16 percentage points, the opposite of what those in the short-term rationale camp might expect. The 95% confidence interval of this point estimate ranges from -0.43 percentage points to 0.11 percentage points. In other words, we don’t have enough evidence to say that the effect of fumbling last week on fumble rate this week is meaningfully different from zero. This doesn’t mean that momentum doesn’t exist from game to game when it comes to fumbling; we just couldn’t find sufficient evidence to support it here.
It might be better to look at this question on a more granular level by using play-by-play data. The play-by-play version of this study might pose a question like “What are the chances of a running back fumbling sometime in his next ten carries after a fumble?” If we use games, a coarser unit of time than plays, we lose information, so answering this question using play-by-play data seems like the logical extension of this project.
Another study could also improve on these methods by using better covariates to identify similar players. We used basic stats like yards and touches to quantify running back similarity, but more sophisticated stats would probably provide a more accurate proxy for a player’s past quality and characteristics. This more advanced set of covariates could help us avoid confounding factors, enabling us to get closer to the true causal effect of recent fumbles on current and future fumbles.
While this study design leaves room for improvement in the future, it provides some evidence against employing the short-term rationale when benching a running back for fumbling. If coaches are really using this rationale, they are actively hurting their teams, because they are sacrificing overall running back quality for a potential gain in turnover margin that we cannot find evidence for. Unless evidence for the existence of fumbling momentum emerges, coaches might want to leave the immediate-fumble-benching strategy behind.
Except Mike Tomlin. Someone needs to provide intro paragraph material for all the football writers out there.
To some extent isn't this testing whether or not fumbles have consistency over a given time frame. It seems that even if there was a statistically significant result, even on a play by play basis, the take away would not be about if benching is a good choice but if certain players just can't hold on to the ball (potentially fumbles are less random occurrences than they get said to be) and if so then is the replacement value of the second string guy worth swapping for.
For this hypothesis (should coaches bench running backs) to be tested wouldn't you want to see if after a running back is benched they are less likely to fumble in future downs. These downs could be in the same game or in later games. Clearly that study would require a much larger data set, including play by play and subjective benching info, which is most likely outside the scope of this study (and the time available for a college senior).
Fun piece though, I have been reading the HSAC blog for a few years and enjoy what you guys put together, congrats on being included on this blog.
It's possible that coaches bench RBs after not seeing improvement in ball security in practice, and leave RBs in if they do see improvement. Then the fumble rate would be uncorrelated from season to season because the RBs who would have had a consistently high fumble rate aren't included in the study (because they get benched).
So the question then is whether coaches are capable of judging that. I wouldn't think so, but the coaches probably believe they are.
Good stuff. I'm wondering if you could model each individual players play-by-play fumble rate after a poisson process (and if so, it would re-enforce the idea that fumbling recently does not necessarily lead to being more likely to fumble soon, quite the opposite). This reminds me a lot of how NBA/NCAAB coaches bench players in foul trouble early in games -- although that obviously has the caveat of being able to foul out, you can't "fumble out".
I also agree with Eli. It would be interesting to see if the benching itself has an effect on future fumble rates.
Based on my limited knowledge of football skills, I think this might be an area that is actually poorly suited to a purely statistical analysis. There's a world of difference between an RB who is using poor ball control and fumbles on an otherwise routine play versus an RB who happens to get a helmet up under his arm and has no chance to avoid a fumble. Benching a guy who's not playing carefully may have benefits, whereas benching the unfortunate is more like zapping rats in a Skinner box. It's not at all clear to me that the play-by-play data is finely resolved enough to distinguish between these two important categories.
How would you test this? I guess you could have some coaches watch tape of a guy prior to a fumble/non-fumble and assess whether good ball-protection is being employed. Then find the correlation between fumble percentage and quality of protection. Follow up by judging whether particular players consistently use good protection. Then finally assess whether players can improve their protection post-benching? Sounds tough, but then I'm not getting paid millions of dollars to analyze football.
I'm a statistical neophyte, but something about this analysis bothers me. If your goal is to assess someone's fumbling rate across multiple seasons doing an autocorrelation looking at year n with n+1 seems arbitrary since you are only looking at correlation between two given time points instead of all timepoints related to that player. Wouldn't it be better to do an interclass correlation? I don't know if that is possible given that players play for a variable number of years. But if a player has a fumble rate of 0.002 in year n and n+2, n+3, and n+4, and a fumble rate of 0.2 during year n+1, wouldn't that appear as noise in your autocorrelation, but may be more likely to be picked up in an interclass correlation?
There is somewhat of a selection effect in the N/N+1 regression, since people who fumble a lot in a given year must not have fumbled a lot the year before (or they wouldn't have been given the opportunity to fumble a lot this year). So if a RBs career ends with a high fumble rate year (unluckily, but unsurprisingly), then there is no subsequent year data that would show regression to the mean and remove the N/N+1 correlation produced by that last, bad year. An even/odd game regression over an entire career (excluding the final year) should mitigate this somewhat.