## 2009 Team-Specific Run-Pass Balance

Recently I've been looking at run-pass balance on first downs based on a principle of game theory. When strategy mixes are optimized, the two strategies will ideally produce equal payoffs. If they aren't equal, then the better strategy should be selected until the opponent responds with his own counter-strategy. Results suggested that, in the NFL overall, the gains by passing on first down exceed those by running. In turn, this suggests that offenses should pass more often than they currently do.

However, every team has its own relative ability between passing and running. You can't just tell the 2009 Raiders to start passing more often. Their running game may actually be superior to their passing game in terms of expected payoffs, so while most teams should be passing more frequently, it's possible a minority of teams should be running more often.

I looked at each team's current run-pass balance in terms of average Expected Points Added (EPA) for each of their first down plays. Again, I limited the data to 'normal' football situations, when the score is within 10 points, and to the first and third quarters when time is not a factor. This way, the effects of 'trash time' and hurry-up offenses are excluded. In this post I also limited the data to plays between the 20-yard lines. Inside the 10-yard line, running actually shows a higher payoff, so for now I want to isolate the pass-friendly part of the field.

Below is a table of each team and their percentage of first down passes through week 12 of the 2009 season. Each team's average EPA for runs and passes is also listed, followed by a total EPA for all first down plays. The final column lists the difference between the EPA for passes and the EPA for runs, which is an indication of imbalance in the respective payoffs of passing and running. Larger positive numbers suggest a team is better off passing more often, and negative numbers suggest running is actually the better choice.

 Team % Pass Pass EPA Run EPA Total EPA EPA Diff ARZ 48% 0.58 -0.13 0.21 0.71 DET 39% 0.26 -0.20 -0.02 0.46 CHI 56% 0.14 -0.31 -0.06 0.45 SD 43% 0.45 0.02 0.20 0.42 MIN 41% 0.36 -0.07 0.10 0.42 IND 55% 0.30 -0.03 0.15 0.33 HST 49% 0.16 -0.16 0.00 0.32 TEN 35% 0.28 -0.03 0.08 0.31 NO 51% 0.38 0.07 0.23 0.30 DEN 46% 0.27 -0.02 0.11 0.29 KC 48% 0.01 -0.17 -0.08 0.17 CIN 48% 0.13 -0.03 0.05 0.17 NYG 54% 0.29 0.13 0.21 0.16 PIT 50% 0.21 0.06 0.13 0.16 WAS 39% 0.04 -0.09 -0.04 0.13 NE 57% 0.11 0.01 0.07 0.10 ATL 49% -0.04 -0.11 -0.08 0.06 CAR 40% 0.02 -0.02 0.00 0.04 PHI 64% 0.24 0.24 0.24 0.00 JAX 59% 0.15 0.17 0.16 -0.02 CLV 35% -0.04 0.02 0.00 -0.06 NYJ 40% -0.19 -0.11 -0.14 -0.08 BLT 47% 0.00 0.17 0.09 -0.16 SEA 53% -0.23 -0.06 -0.15 -0.17 DAL 48% 0.03 0.25 0.14 -0.22 SL 41% -0.26 -0.02 -0.12 -0.24 BUF 47% -0.18 0.08 -0.04 -0.26 GB 48% -0.29 0.02 -0.13 -0.31 SF 44% -0.21 0.16 0.00 -0.38 MIA 32% -0.35 0.17 0.00 -0.52 OAK 45% -0.46 0.10 -0.15 -0.56 TB 46% -0.68 -0.03 -0.33 -0.65 Avg 47% 0.04 -0.01 0.02 0.05

About two thirds of teams have an imbalance that favors passing (positive difference), and one third of teams have an imbalance that favors running (negative difference). For example, Arizona tops the list of teams that should pass more. Their combination of a strong first down passing attack and anemic running game suggests they might want to increase their passing percentage well above their current 48%. At this point in their strategy mix, every run they call costs somewhere between a half and a full point on the scoreboard. Surprisingly, Detroit is second on the list, partially due to a woeful running game on first down.

At the other end of the spectrum is Tampa Bay. Their first down offense is below average both in the air and on the ground, but their passing game is especially woeful. Every dropback on first down is costing them over half a point. You might think this means they should run much more often, but they are frequently underdogs and may need passing's high variance outcomes plus some luck to win. But other teams with both a negative imbalance and a positive running EPA, such as Baltimore or Dallas, should consider running more often.

The sample sizes are relatively small here, ranging from 83 plays for the Raiders to 129 plays for the Steelers. I'd expect these numbers to regress significantly over the long run, and statistical significance is definitely an issue. However, we're not trying to declare a scientific truth here. Teams have to make a decision about their strategy mix based on the information available to them, and this is all we've got at the moment.

This is just a rough first sketch regarding what can be done with this concept, and there is more that we can do here. We can increase the sample sizes by including the early parts of the second and fourth quarters. We could also regress the numbers to an appropriate degree for a more realistic estimate. With better data, we can classify passes as deep, short or screen, and figure out what kinds of passes are helping and hurting--and do the same with run types.

Ultimately, play calling "balance" shouldn't be a league-wide commitment to run an arbitrary percentage of the time. Each team needs to balance its play calling by the weights of its own strengths and weaknesses.

### 24 Responses to “2009 Team-Specific Run-Pass Balance”

1. Dan Rosenberry says:

There's an embedded assumption that may not be true in this analysis. Is the marginal EPA always constant? If it's not then average EPA doesn't equal marginal EPA.

Teams should work to make the marginal value of the strategy choices equal, not the average value, so that at the margin they're indifferent.

It's possible to construct marginal EPA by play call % curves that justify any conclusion by modifying the shap of the curve. Without knowing which calls are a teams marginal runs and marginal passes (they almost called the other type of play) it's hard to compare to see if the value of the strategies is balanced at the margin or not.

2. Ian says:

Dan

Just so I can get the idea of marginal versus average correct in my head, can you confirm that what follows is what you're saying?

Let's assume I'm the head coach at Detroit (presumably I sinned in a past life) and I've run 100 first down plays this season. My 39 passes scored 0.26 EPA and my 61 rushes scored -0.20 EPA, therefore Brian's game theory suggests that I ought to pass the ball more often. But what you are saying is that if, after my 39 passes and 61 rushes, the expected EPA (that is, the expected 'expected points added') of my next play is the same regardless of whether I rush or pass, then my strategy would be balanced at the margin.

If that's the case then I would expect that past performance is a reasonable predictor of future performances (although I have no stats to back this up) and that the average EPA is the same as the marginal (I don't know how one would measure/prove that though).

3. alex says:

this kind of analysis is really really awesome, look forward to more.

4. Brian Burke says:

The marginal vs. average debate is mostly semantic here.

What 2-player zero-sum game theory says is that the more a strategy is selected, the more often an opponent will counter it with his own strategy choices. Therefore, every additional choice of a given strategy will reduce the expected payoff of that strategy while enhancing the payoff of alternative strategies. That's all part of game theory.

Note that I'm not saying that, for example, the Cardinals would add 0.7 points for every future pass they throw on first down. I've said that every past run they've called has cost 0.7 points. I've also qualitatively stated that passing more often would benefit the Cardinals. They should theoretically pass more often until the payoffs equalize, but exactly how often that is and how much it will benefit them remains to be seen.

5. Dr Obvious says:

Please correct me if I'm wrong, but won't we be able to find the optimal run/pass mix with the equation (assuming both run and pass EPA are possitive)
Pass EPA(1-Optimal Pass %) = Run EPA(Optimal Pass %)
Using SD, that comes out to 95% pass on first down. A few more:
Pit = 78%
Philly = 50% (note they have the same EPA's)
Dal = 11%

Or I guess that would only work as a starting point, and as defenses adjust, the EPAs will change and you should track this and adjust.

If this is right though, I'm having trouble wrapping my head around why it is that equation, and not:
Pass EPA(Optimal Pass %) = Run EPA(1-Optimal Pass %).

6. Anonymous says:

Dr. Obvious - I think you're failing to account for their current mix of Run & Pass. E.g., Philly's play-calling is optimal based on passing 64% of the time.

Funny that the "optimal" play-calling mix is from Philly. Reid and/or Mornhinweg have probably taken as much criticism as anyone for neglecting the "run to set up the pass" myth. Turns out they are the model of efficiency!

As for Reid's game management .... that's another story.

7. Anonymous says:

THINK ON THE MARGIN

If, for example, passing yields more yards or estimated points per (first down) play, you don't keep passing until your running and passing averages are the same.

You keep passing until your expected value ON THE NEXT PLAY is equal. This will be different from your average of the last X first downs.

8. Dr Obvious says:

Anon 2 up - Yeah, right. That's where my error is. Thanks.

9. Jonathan says:

"Funny that the "optimal" play-calling mix is from Philly. Reid and/or Mornhinweg have probably taken as much criticism as anyone for neglecting the "run to set up the pass" myth. Turns out they are the model of efficiency!"

Wow, good call.

10. Anonymous says:

Is the possiblity of a turnover considered in the EP calculations?

11. Anonymous says:

Jonathan - that is only 2009 for the Eagles and not indicative of previous seasons. Additionally, the Eagles HAVE run the ball more this year as evidenced by the last two games. We are talking about ridiculous situations where the Eagles keep passing in 80-20 ratios in games that are within 10 points. The Eagles are 7-4...the best record they've had since 2004. Moreover, this also doesn't account for advantages in controlling the clock when running.

12. Brian Burke says:

Yes, turnovers, penalties, and everything else are factored in.

13. Dan Rosenberry says:

Ian-

What I meant was that a pass is not the same as any other pass. Hail-mary's are utilized between 0 and 10% passing, and line plunges on ""and inches"" are part of 0 and 10% running. Brian's talking about first down calls, so that's less of an impact here (I missed it completely)

------
Commenting on marginal vs average as it applies to first downs.

The passing play between 50% passing and 60% passing might not be the same mix as between 0% and 10% (fewer screens in 0 to 10% for instance, long bombs are thrown first, etc). If it is the same, then the average EPA is the marginal EPA, if it's not then it's not. I can't think of any way to quantify this, but it is an assumption of applying the game theory model.

Another part of this is ""taking what the defense gives you"" with a flexible quarterback that audibles (it's not just paper/rock/scissors). The marginal run call might be against a fairly stacked/obvious pass defense, and the passing EPA in that situation would be significantly less than the average, even less than the low running EPA, causing run to the be correct choice.

If the defense plays pass defense 45% of the time and Indy is smart enough to recognize all those instances, 55% passing (on every down that the defense plays run) might be their best response.

My point was that the game theory assumption about a pass being a pass being a pass might not be an accurate depiction. The root question is what the marginal pass and marginal run looks like and if they're the same as the average pass/run, the result (pass more) holds, if they're not, it isn't clear. I can't think of any way to figure out which passes the team almost called as runs versus which runs they almost called as passes without inside information.

14. John H says:

Does it matter that there is almost no correlation between optimal play calling and team success (a coefficient of -.036 using the absolute values for EPA diff and each team's winning percentage)? Obviously, other factors contribute to wins and losses, but if play calling is a crucial component to success, wouldn't you expect the teams with the best records to have a close to optimal mix? 4 of the 6 teams with the largest positive EPA difference are almost locks for the playoffs, whereas most of the teams with the lowest absolute EPA difference are pretty much hovering right around .500 records. It is only the teams with the largest negative EPA differences that appear to be struggling on average.

Which brings me to the next point. The assertion that "offenses should pass more often than they currently do" seems to be refuted by the fact that teams with positive EPA differences have a combined record of 113-85, while teams with negative values are 57-88. If anything, doesn't this suggest, in general, teams should be running more, as run-heavy teams are more successful?

I am sure there are plenty of underlying factors which explain play calling mix (for instance, the point about teams like Tampa Bay being frequent underdogs), and I agree with the general principle behind your argument, but it just seems like there should be a relationship between team success and play calling. Any ideas why there is not?

15. Brian Burke says:

John-You're looking at 3/4 of a single season here. For all teams 2000-2009, passing more often on first down correlates positively with greater overall EPA (0.14), conversion % (0.18), and average gain (0.27).

To me this means that pass-heavy teams are doing better, and they would do even better if they passed even more.

16. Brian Burke says:

And true, as I always say 'the usual caveats apply,' which means not all passes are the same animal. We don't know which kinds of passes in what kind of mix might be optimal, but we do know one thing: they're not runs!

17. Anonymous says:

Avg. v. Marginal

I notice you've taken the word "average" out of your article, but you're still talking about averages. But you've only made the language ambiguous and did nothing to the data. Intentional equivocation.

18. Anonymous says:

...if still accidental equivocation.

19. Brian Burke says:

Dude-I have not changed the article at all. No equivocation, intentional or otherwise.

The guys who are still debating the marginal vs average semantics in a 2-player zero-sum game aren't in an argument with me. They can take up the topic with John von Neumann.

20. zlionsfan says:

Re Detroit: perhaps part of the problem is Detroit's predictability on first down.

According to Football Outsiders, the Lions' DVOA on 1st down run plays is -15.0%, 30th in the NFL, and their DVOA on 1st down pass plays is -22.6%, 31st in the NFL. Note that these numbers include all first-down plays, not just those that you looked at ... so it's possible that in "normal" first-down situations, the Lions throw the ball better than they run, and in situations where they trail by a lot, the opponent is expecting pass and the Lions' lower success might drop passing below running. (For the record, the only down/play combination in which the Lions succeed is in 3rd/4th-down running plays, 42.2%, 5th in the NFL, and I wonder if that's also a function of the games in which the Lions were so far behind that it didn't matter if they converted a third-and-short.

21. Brian Burke says:

Yup. Once any team falls behind by a lot or time gets tight, their plays are almost all going to be passes, including first downs. The Lions unfortunately find themselves far behind early in games and become very predictable.

22. Dan Rosenberry says:

It isn't an argument with von Neumann, it's a suggestion that the model for football might include information about what the other team is doing (personnel choices). Sorry about not being clear that I was picking at the the model, rather than the result.

Specific example of how marginal play EPA could matters
--------------------
The game has an extensive form where the defense lines up, the offense makes a read as to the type of defense and then makes a play call. The offense three different reads on the defense, run stack, unknown, and pass stack. In that order, the payoffs for passing are 0.36, -0.3, -1, and the payoffs for running are -1, -0.18, 0.09.

When the defense can be decoded the payoffs indicate calling the other type of play. When it's unknown, it's worse to try passing than to try running.

Assuming the defense plays 50% run stack, 25% unknown, 25% pass stack, this payoff structure is consistent with the observed datapoints for the Colts. The Colts pass 55% of the time, all the run stack defenses and a bit of the unknown defense, average EPA is 0.30. The 45% runs are called on the pass stacks and most of the unknown defenses, average EPA is -0.03.

However the marginal play is against the unknown defense, where the payoffs suggest that the Colts should run more, not less, despite the observed average passing EPA being higher than the observed average running EPA.
------------

Just because the value for one strategy is higher than for another doesn't mean it should be called more often, if the choices for the participants are interrelated.

23. Z-Dog says:

A point worth mentioning is that the underlying factors that make running or passing more successful are in flux. Pittsburgh is a better running team with Mendenhall and Dixon than they were with Parker and Roethlisberger. NYJ look like they should run a lot more, but some of those numbers came from a healthy Leon Washington, and a QB who threw a ton of picks. As Sanchez gets better at ball security, the expected value of passing should rise.

I think coaches probably look at things more in this way. They have a philosophy about their ideal offense and its run-pass mix at different points in the game, and they seek to construct and coach a team that can produce those outcomes.

24. Mark M says:

Dan Rosenberry...

But the data set excludes hail marys, plunges etc (it's all first down plays in the first and third quarter, between the 20s with the game within 10 pts). They won't skew the points. This is the 'normal' football situation he speaks of.

I suppose you've a point about the model assuming play-action, screens and standard drop-back passes all being the same (it would also class a QB scramble as a run rather than a called pass).

I think Brian has it correct with the wording above e.g. so far this season Arizona has done better passing on first down than it has running, and would likely benefit by increasing the number of times they pass on first down.