Another Run-Pass Balance Study

Benjamin Alamar, author of the Passing Premium paper (critiqued here), takes a second stab at comparing the values of running and passing with new research. A brief presenting his methodology and findings was presented at a recent symposium on sports statistics. This time Alamar uses expected points as a measure of value, and compares the EP values gained by passes with those of runs. He defines risk as the probability each play type will result in a negative EP change, and finds that running is both less productive and "riskier" than passing. There are three fairly big problems in the methodology. Fortunately, all three can be rectified.

First, Alamar creates his EP values using a simple linear regression (see slide 10). Using down, distance, and yardline (plus other variables controlling for quarter and other effects), he produces an EP equation. This is a really bad way to estimate EP values. They're not linear, and there are any number of interacting effects within. The Levitt-Kovash paper, in contrast, uses a regression with quintic terms and full interactions to create their EP values. (I prefer a  more direct method--looking at the data empirically and smoothing it using a method called LOESS.)

Slide 11 shows the results for 1st and 10 situations. It looks like an exponential function and doesn't resemble the real EP curve for 1st and 10s. Romer, Levitt-Kovash, myself, FO all pretty much agree on the shape of the curve, but Alamar's is way off. It doesn't even make sense given the  regression results from slide 10. There's no way to get a curve like that with the regression table he presents, so there's something missing in there somewhere. (I haven't seen the full paper, if there is one, so maybe there is some explanation.) To fix this, he needs to redo how he gets his EP values. There are any number of ways; he could even borrow them from FO, me, or even Levitt and Kovash.

The second problem is the same flaw that plagues the Levitt-Kovash paper. Time and score are very important considerations in the value of any play. The value of a run to a team ahead can be far more than just the EP advantage gained. Running time off the clock can be just as valuable, if not more valuable. Additionally, the advantage of a low-variance outcomes needs to be considered. The regression used to create Alamar's EP values does not does not account for the time-value of plays.

There are two possible solutions to the time/score problem. One possibility is to use win probability instead of EP. WP is a measure of utility that properly accounts for time and score in addition to down, distance, and field position. But a WP model is extremely difficult to produce, and any model is going to have thin spots where a lot of assumptions need to be made. This is true particularly on 2nd and 3rd downs when to-go distances are variable.


The second possibility is to restrict the analysis to "normal" football situations. By normal I mean early in the game when neither team has a big lead. This is when teams are (or at least should be) playing at their optimum risk-reward balance for net point maximization. With enough data, there should be no problem restricting things to the first quarter when the lead is 10 points or less.

The third problem has to do with his definition of risk--the probability a play will result in a negative change in EP. Suppose I want to cross a busy street, and I have two strategies: I could wait patiently until traffic clears, or I could run wildly across as fast as I can. Waiting patiently has a very high probability of resulting in a negative utility for me. My time is precious, and waiting can be costly. On the other hand, running wildly across the street results in no wait, plus there is a still relatively low probability of a negative outcome--which in this case would be being hit by a car that doesn't stop for me. Given this definition of risk, running wildly would be the safer strategy.

The problem is that the size of the negative outcome for each strategy is vastly different. The  majority of runs are actually negative in terms of EP. Runs that result negative EP changes include up to 4-yard gains. Incomplete passes would also be negative, but so would pass plays with the added risk of interceptions and sacks. The costs of negative passing plays are disproportionately higher than for runs. The full weighted distribution of consequences needs to be considered.

Defining risk this way leads to some conclusions that don't make sense. Consider an offense with a small lead facing a 3rd and long deep in its own territory. A draw play is considered the epitome of a low risk play. Passing for the first down is the risky thing to do because of size of the cost of an interception or sack. But the probability of a negative for a run is much higher than for a pass because any gain shy of a 1st down would be deemed negative. In terms of the probability of winning the game, the run is the less risky choice.

My recommendation is to do three things. First, get a different table of EP values. Second, limit the data to "normal" time and score situations. And third, redefine risk that takes into account the sizes of the costs of negative outcomes.