Before tools such as EPA and WPA were available, I relied on team efficiency stats to estimate team strength. Yards per pass attempt or per run attempt worked out to be very good estimators of how good a team was, especially if ‘good’ is defined as being likely to win forthcoming games. Efficiency stats had the added benefit of being relatively simple, widely available, and easy to calculate.
Efficiency stats also worked well in regression analysis. In a regression model, it’s best if the predictor variables are independent of each other. In other words, the less each predictor variable correlates with the others, the more valid and reliable the resulting model will be. Passing and running efficiencies in the NFL correlate weakly. Over the past 10 seasons, offensive passing and running efficiencies for each team correlate at 0.09 (where 1 would mean lock-step correlation and 0 would mean complete independence.)
Problems with efficiency stats
However tidy independence makes a regression model, it defies the logic of game theory. In football speak, it would mean there is little evidence that “the run sets up the pass” and vice-versa. In zero-sum two-strategy games, outcomes are optimized at an equilibrium (called the “minimax”) where the benefit of one strategy equals the benefit of the other strategy. Although this is provable mathematically, it makes intuitive sense too. If one option pays off better than another option, a player should choose the superior option more often until his opponent responds appropriately. Eventually, the payoffs will equalize at a stable payoff. Any deviation by either player from the optimum mix of strategies opens a hole for his opponent to exploit. Consequently, we should expect a reasonably strong correlation between passing and running efficiencies. There are limited examples of football coaches “playing minimax” in situations when the payoffs are clear, but for the most part, that’s not the case. So what exactly are coaches optimizing? Why aren’t passing and running efficiencies more connected?
Raw efficiency stats also point to the futility of the running game. Offensive running efficiency correlates with team wins at 0.15, a meager relationship compared to the 0.66 correlation of passing efficiency. This stark difference suggested that teams are investing far too much attention and resources into running the ball. My theory was that as the pass become more successful, with efficiency climbing and interceptions plummeting over the recent decades, coaches were slow to catch on, clinging to outdated convention.
New Tools
With new tools and new stats however, it's now clear that the relative futility of the running game is overstated by looking at efficiency stats alone. Chase Stuart of Pro-Football-Reference.com suggested that running efficiency might be too sensitive to long runs, which tend to be rare and relatively random. So I took a fresh look at things and found some very interesting results. It turns out that running correlates with winning to a much higher degree when we look at it using Success Rate (SR).
SR is a very simple concept in principle and has been around for decades. Each play is graded as either a success or not based on its outcome. For example, if a play gains 3 yards on 3rd and 2, that would be a success. But those same 3 yards would be a failure if the situation were 3rd and 4. In the seminal book from the 1980s Hidden Game of Football, the authors devised a simple rule of thumb based on their intuitive sense of football success. A success would be: On 1st down--a gain of 4 or more yards; on 2nd down--a gain that at least halved the distance to go; and on 3rd down--a conversion for a new set of downs.
Although I have nothing against this rule of thumb and am generally a big fan of simplicity, my own definition of SR is different. Using the concept of Expected Points Added (EPA), successes can be defined more precisely. Any play that results in a positive change in an offense’s net point expectancy can be considered a success. This technique not only accounts for down and distance considerations, but for field position as well. For example, a play that gains 10 yards on 3rd and 12 would normally be considered a failure, but if those 10 yards put the team in field goal range, that might be considered a success. It also accounts for the effects of the shortened field in the red zone.
Importance of run Success Rate
Based on SR, running correlates with team wins at 0.40, much higher than the 0.15 correlation based on efficiency.
SR reveals an even more interesting revelation. I think it finally answers the question of what coaches are optimizing. Based on SR, passing and running correlate at 0.41, a much stronger relationship than the 0.09 correlation based on efficiency. Coaches are optimizing Success Rate, although they’re probably thinking of a simpler version of SR, much more similar to the rule of thumb than to my EPA-based definition.
So NFL coaches are playing minimax after all! They’re just using a very simple payoff function for the value of each play—either success or failure. The correlation between offensive run and pass EPA is smaller than for SR—0.35. This is remarkable because it suggests that coaches are not as sensitive to the magnitudes of the payoffs as they are to the simple dichotomy of “success.” This is understandable, because without an EPA model running in your brain, it’s impossible to accurately assess the value what the myriad of possible outcomes on any given play. Coaches are human, and the easiest and simplest way to value outcomes is to say, “Yeah, I think we’re better off than before,” or “Nope, that didn’t work.”
This mindset is echoed in an old football saying, attributed to Texas coach Darrell Royal and still often repeated: “Three things can happen when you throw the football, and two of them are bad.” That says it all, doesn’t it? Coaches classify outcomes as good things and bad things, and then count them up. I’m sure coaches obviously know that interceptions are a lot worse than an incompletion or a 1-yard stuff. But they don’t accurately account for the payoff and frequency of each possible outcome.
As I understand it, most coaching staffs grade player performance the same simple way. For every snap, players are graded-out either as successful or not, and their total for the game is compiled into an overall percentage grade. It should be no surprise that coaches think of entire plays in the same way.
Whether coaches admit it, or could articulate it or are even aware of it, they're predominantly thinking in a simple success-or-not paradigm.
Defensive Success Rate
Defensive SR shows similar relationships as offensive SR. Although defensive run efficiency correlates with team wins about the same as defensive run SR (-0.16 and -0.17, respectively), the correlation strengthens to -0.26 when we only look at runs from the first three quarters. This excludes the highly predictable run-out-the-clock runs in the 4th quarter that are usually short gains. Additionally, defensive run and pass SR correlate with each other at 0.41, stronger than the 0.31 for defensive run and pass efficiencies. Just like on offense, net pass efficiency is still king on the defensive side of the ball, correlating with team wins at -0.56. (Defensive stats usually correlate with winning negatively, because lower numbers are better.)
Implications
Coaches appear to be overly focused on play-level success (represented by SR) and not focused enough on drive-level (represented by EPA) and game-level success (represented by WPA). They’ll spend late nights in the film room dissecting every possible match-up for the slightest advantage on a single play, but they’ll ignore the numbers that suggest they pass more or go for it on 4th down. They’re looking down at the sport from a 10-foot ladder when they should also be looking at it from the 10,000-foot level.
It’s possible that SR can help improve prediction models that currently rely solely on efficiency statistics. I’ve already tested several SR-based regressions, but for one reason or another, they fail to outperform the efficiency-based models. The problem is more complex with SR, because of the correlation between running and passing. Ideally, run and pass SR would have equal weights because of game-theoretic effects, and overall team SR would be the only thing that matters.
It’s also notable that passing efficiency, both raw efficiency and passing EPA, remains the most important facet of the game. No matter how it’s sliced and diced, passing reigns supreme. However, running is more important with regard to winning than previously indicated, and it does help “set up the pass” as game theory had predicted.
Does this mean that teams should be running as often as they do? No, not in most circumstances. There is a difference between descriptive and prescriptive analysis. This is descriptive, which means this is what coaches are doing to win games. If all the coaches adhere to the same conventional wisdom, their strategic flaws would virtually never be exposed. The prescriptive analysis remains the same. Generally, teams should be passing more often on 1st and 2nd down, and running more often on 3rd down and in the red zone.
The bottom line is that we should pay attention to run SR. It’s very likely a predictor of passing performance and of overall team success.
How Coaches Think: Run Success Rate
By
Brian Burke
published on 10/04/2010
in
modeling,
research,
run-pass balance,
strategy
Subscribe to:
Post Comments (Atom)
I was thinking about the run pass equilibrium yesterday and realized that while an epa balance makes sense, an efficiency balance doesn't seem to tell you much. As the extreme example if you have a 97% chance of running for between 3 and 4 yards per play you get about 3.5 yards per attempt. Then assume you average 6 yards per pass play but you have a huge spread from zero to 95. With out running the numbers, I assume the winning strategy is to run on every play. I assume that even in the higher variance world of the NFL there is a lot to be said for this.
Has anyone noticed that Josh McDaniels seems to be the most aggressive coach on 4th down, even though his team with about 6 rush yards per game is probably the least likely to convert? I assume the decisions are still correct, and the other coaches are just that dull.
IHMO, SR accurately explains almost everything that coaches do (and most of what we complain about on sites like this). For example, in your excellent analysis on 2-and-short attempts, you showed that this is really a "free" down - a time when offenses should be thinking big and swinging for the fences. And yet, the primary call on 2-and-short is a simple run that is likely to achieve "success" (a first down) but extremely unlikely to achieve a true success (a score, or at least, a long gain).
Very interesting look at things. It almost suggests that a coach would consider a 5 yard run, followed by a 3 yard run, followed by no gain and a punt (2 out of 3 successful plays) as more of a success than two incompletions followed by a 12 yard pass (1 out of 3). And actually watching games that doesn't seem too far from the truth.
There's also the matter of blame. A coach who calls three passes in a row gets the blame if they don't result in a first down - whereas failure to convert a 3rd and 2 puts the blame on the players.
On the matter of 4th downs, there was a good one in the Jets game yesterday. Facing 4th and 1 on their own 31, the Jets seemed to consider it before punting. The commentator said something like "if you have confidence if your defense, you give them a long field to defend". That made no sense to me - if you have confidence in your defense then you should be happy giving them a short field. Of course, the punt was then returned 26 yards by Roscoe Parrish which meant the Jets gained 18 yards of field position for their defense by punting it.
Excellent post. I learned from playing poker (and reading) that your job isn't to do what's most likely to win the hand--you're supposed to do what's most likely to win the game. Most people focus on the SR, or the short game, whether it be poker, football, pool, or politics.
There's always a bigger game.
Josh McDaniels is definitely paying attention to the math.
Brian,
Wondering if you looked at the correlation between pass proportion (i.e. pass plays / total plays) and pass success rate?
Does success rate explain the focus on run/pass balance? Does the pass success rate decline as the pass proportion increases (understanding that causality runs both ways, a high pass success rate can cause the pass proportion to decline).
The other element that plays into this is the game theory element of run defense vs pass defense called.
What is the yardage to go where a given opponent has a 50% probability of calling a pass play vs a run? Obviously that changes by down, time to go and score. The Indy game earlier this year when the opponent played nickel the whole game did a good job of slowing down Manning's passing game. They did pay a price.
I don't know if it is possible, but if you could tap into the data you used for the run vs run and efficiencies for each of the 4 run/pass cell, I think it would yield some interesting insight.
running efficiency might be too sensitive to long runs, which tend to be rare and relatively random
I've long believed the "great break away running back" not only is greatly overvalued but is maybe a curse, because teams overinvest in building an O around him and compromise their passing games. From Gayle Sayers to Barry Sanders. The exciting "long plays" those backs produce can never be as reliable or worth as much as the conventional passing plays sacrificed.
SR reveals an even more interesting revelation. I think it finally answers the question of what coaches are optimizing ... Coaches are optimizing Success Rate...
the mindset is echoed in an old football saying, attributed to Texas coach Darrell Royal and still often repeated: “Three things can happen when you throw the football, and two of them are bad.” That says it all, doesn’t it?
This is moving things back in the direction of a discussion in the early days of your blog, about whether coaches try to get first downs instead of yards ... whether the most meaningful running stat is median yds/carry (or even the mode) rather the average, since a few break-away long gains can produce a superior average masking an inablility to reliably make a 3rd and 2 ... how risk/reward affects run/pass mix, since the much higher risk of several big loss plays occuring in a bunch in the passing game can more than offset the higher average per play it produces in the long run. (Even Peyton threw 6 picks in a game a couple years ago.)
The extreme model for high success rate/low risk is a team that runs for 3.5 yards each and every play. It is an unstoppable scoring machine, and will crush a passing team that averages 10 yards per play but with 40% incompletions.
The real life prototype of that of course was Lombardi's Packers. Take the 1966, 14-2 SB I champions. By efficiency, yards/play-wise, they were a terrible rushing team, 3.5 per carry, tied for 13-14th in a 15-team league. But as bad as their running game was they were near top of the league in running attempts and dead last in pass attempts.
Which indicates, efficiency-wise, their coach was from bizarre to stupid, because their passing game dominated the league. Starr had an AYA of 9.6, a full 1.6 over the next guy, and a TD-Int ratio of 14-3, which would be tremendous today and was *fantastic* in that era when the ratio for the rest of the league was 27-32. To keep running & running with such a poor running game, while passing less than anyone else with the league's dominating passing attack ... how *dim* was that coach??
Of course, anybody who said back then that Lombardi's Packers had an awful running game would have been guffawed out of the room. They had a crushing running game. Success rate was what it was all about. I haven't seen play-by-play for 1966, but Lombardi's bios discuss these numbers and detail how the "Lombardi sweep" was explicitly designed to get 3-4 yards each and every time, and forget the longer gains. With few longer gains, its average/run was very low. It's first down conversion rate was very high. And the risk incurred by the whole O via turnovers and big losses (such as sacks) was league-bottom. So they were never going to happen in a bunch.
"Success rate" was the name of Lombardi's game, embodied in a a simpler era.
I was working up a similar analysis for your community, but I guess you've made that moot.
It's good to see you finding a resolution to this "paradox". It's been hard for me to accept that NFL coaches grossly "under pass", because while in very competitive environments it is easy to find small inefficient aberrations when there is an explanation for them (e.g. coaches' reluctance to go for it on 4th and 1) it is very hard to find any large continuing systemic failures that persist with an only apparent explanation of "universal stupidity" even after the remedy has been tried.
The idea "teams pass too little" is hardly new, and led to the Run and Shoot (or "chuck and duck" as Buddy Ryan called it) getting a full-scale NFL tryout 20 years ago. And it was always tremendously successful at running up yards and points -- proving "for more yards and points, throw more!" Just like those efficiency stats say. Remedy found.
For coaches afterward to universally forget that and go back to throwing too little, for no apparent reason, would be inexplicable univeral stupidity.
But the R&S disappeared as a base offense, and became a situational adjunct, after Houston blew that 35-3 lead against Buffalo in the playoffs, the worst fall-from-ahead collapse ever. Houston's problem with its R&S that game wasn't that it couldn't score but that it couldn't reliably (or even semi-reliabily or sorta reliably) convert a first down and run off some clock time as it was blowing a 32-point lead in the second half, with 22 pass attempts to 4 rushes. Zero success rate when it counted.
Which is closely related to the risk-reward of passing. The real risk isn't that over the course of a game you'll have some incompletes, sacks and picks, that are compensated for by higher yds/play in the long run. It's that you'll get a whole bunch, maybe a dozen such plays in a row, that will kill you dead because there is no long run. A football game is only about 60 offensive plays long and gets shorter with each one. So "negative success rate, compounding" is a real danger with too much passing. It can change 35-3 to 38-41.
But if coaches are calling plays to equate the positive success rates of rushing and passing, then moving back from the R&S to pass less often is explicable, they are acting reasonably rationally and there is no great mystery in their behavior to explain.
"The extreme model for high success rate/low risk is a team that runs for 3.5 yards each and every play. It is an unstoppable scoring machine, and will crush a passing team that averages 10 yards per play but with 40% incompletions."
I think this is the heart of the issue. I believe that most coaches in the NFL have read something similar to this and they agree with you. I think that is a big part of the problem. The issue is that this is indeed an extreme example and that when you make adjustments for the real world, you very quickly go back to the conclusion that passing is far superior to running overall.
The trouble with the example is the "each and every play" part. Rushing is indeed far more consistent than passing but its consistency is nowhere near what it needs to be to make up for the big play potential of passing. Obviously, you aren't going to rush for 3.5 yards every play. How often do you have to be able to do that to succeed?
Lets say we have two teams. They both get the ball on their own 30 yard line for every possession.
Team A rushes every down. 95% of the time they get 3.5 yards. 5% of the time they get 0 yards.
Team B throws every down. 50% of the time they get 14 yards. 50% of the time they get 0 yards.
Who wins?
Well in order to get a touchdown, Team A needs to have 20 straight successful plays. Even at a 95% success rate, they will still get a touchdown only 36% of the time.
Team B will be nowhere near as successful on a per play basis, but they still only need to complete 5 passes without missing 3 in a row. The odds of them missing 3 in a row at 50% a piece are 12.5%. The odds of them making all 5 passes given 3 shots a piece at each pass is 51.3%.
Team B scores 51% of the time. Team A scores only 36% of the time. This game is going to end in a blowout. Even if you lower the success rate on passes well below 50% and keep the success rate on runs at an absurd 95%, you still end up with passing being preferable.
I think that Brian is really on to something here. The reason why coaches don't pass more is because runs are more likely to be "successful" but that really has far more to do with a fairly odd and rigidly binomial way of defining success than it does with anything to do with actually winning football games.
Ian is perfectly right on. A team can run the ball 3 straight times for two successes and one failure and still end up with the same three-and-out that a team gets from three straight incompletions. Somehow the former isn't viewed as nearly as big a problem as the latter.
The 1992 Houston Oilers are a perfect example of the old expression "The exception proves the rule". I agree that they threw the ball a little bit too often and that they should have exercised a little better clock management. However, I'd point out that was one game from 18 years ago and there have literally been hundreds (if not thousands) of games since then when teams have suffered from too many runs when passes would have been better.
I was also trying to figure this out and I was trying to see how running vs passing on first down affects the liklihood of making another first down (or scoring a touchdown) . Or in other words are coaches too focused on moving the chains rather than maximising yards or points which could mean that they are playing minimax but as Brian says looking at the wrong payoffs. I thought I had an answer but then realised that I had errors in my spreadsheet trying to analse this and I havent had a chance to go back.
James
The thing with Success Rate could be that it is more of an explanatory stat than a predictive stat. Which correlates better with future SR; Yards Per Carry or SR?
I think the deal is this: EVERYBODY knows that gaining (or stopping) first downs is the 2nd most important thing in football (the most important is points, duh). So OC's are most likely to call plays that maximize their "success rate"--getting that next first down. Even a "pass-happy" coach like Payton or Peyton will call fewer passes when trying to bleed the clock in the 4th Q--and the ones they call will be simple, low risk "bubble screens", "flares", etc. to minimize incompletes while maximizing first down opportunities.
So, while a Mike-Martz offense will do very well (provided they have a decent O-line, Chicago) over the long haul, there is a point where a long-term superior strategy can be a short-term failure.
I don't play poker, but to me the best poker strategy would be "what is the best play to win THIS HAND." How will you win the whole pot by losing individual hands? [I understand that over dozens of hands over multiple hours, that being say, a more aggressive bettor/calling more hands/etc. might be better. But if you make the call because it's the best play "mathmatically", and lose the hand, that's still the best play to win the hand--the "river card" just was unlucky for you. If you made the call with a 20% chance to win the hand, and get that ace to turn your 2 pair loser hand into a full house winning hand, that's NOT the best strategy to win--that's LUCK.]
Regarding the question from the PIT-BAL thread, whether PIT should have run or passed on 3rd & 10 from their own 3--I would say that passing gives them a (slightly) better chance of converting the 1st down, but the negative consequences are far worse--an incomplete stops the clock, giving BAL a better chance to score the game-winning TD, and an INT is EXTREMELY BAD--then there is the possibility of a sack, giving BAL the 2 points and the ball. Even a completed pass is not guaranteed to get that 1st down needed.
However, a run, while a very low chance for success, has a reasonable chance of picking up a few yards in a low risk manner, while keeping the clock RUNNING--ESPECIALLY when you have a defense of PIT's quality, and Charlie Batch at QB instead of Big Ben. On 3rd and 10 from your 3, I am pretty sure EVERY play in the book has a low success rate, but if you are ahead late in the 4th Q, I would think that minimizing risk becomes more and more important than maximizing WP [calling Joe Pisarcik].
The poker analogy is a good one. One of the toughest concepts for new players to pick up on is pot odds.
" If you made the call with a 20% chance to win the hand, and get that ace to turn your 2 pair loser hand into a full house winning hand, that's NOT the best strategy to win--that's LUCK.]"
It is the right strategy if you had pot odds to make the call. If there is $100 already in the pot and you need to pay $15 to see the last card, you should do it even if you know that you only have a 20% chance of winning with it. It may seem like luck when you get the last card (and to a certain extent it is) but if you have the right odds, you are making the proper strategy by calling.
By the same token, your opponent is making a huge mistake in keeping the bet small enough that he's giving you pot odds to call when he suspects he has the best hand.
Obviously there are other considerations (how do you know if you have the best hand or not, etc) but the underlying concept is a pretty easy one and I'm amazed at how many times I hear bad players say things like "You just get lucky. I never call a bet unless I thought I had the best hand".
The football analogy is a good one. I get the impression that coaches really don't get the concept of pot odds even though they are being paid millions of dollars to understand game theory.
Put simply, in the end nobody gives a damn if a play was successful or not. You win by winning games and not individual plays.
Running plays are more likely to be a success but the payoff is so small that you need many successes to make up for a few failures. Passing plays are more likely to fail, but the payoff for success is larger so its acceptable to fail more often.
Another thing I realised about success rate, it also brings in the concept of people not valuing gains with the same weight as equivalent losses. Success rate counts a 20 yard gain with the same weight (although opposite sign) to a 1 yard loss. On any one play, yes you'd prefer a success to a fail. But in the long run, net payoff is what matters more than success per play.
Taking the poker analogy, you don't mind giving up small amounts of chips often so long as occasionally you can win big. It can be worth calling often before the flop if it's cheap even if your cards aren't great, because if you get a favourable flop you can win big. If you'd folded you'd have kept your money, but given up the chance of a win.
"By the same token, your opponent is making a huge mistake in keeping the bet small enough that he's giving you pot odds to call when he suspects he has the best hand."
Not quite. By raising the pot, you give yourself an 80% chance at $130, as opposed to a 100% chance at $100. You increase your expected payoff by 4% if he calls.
If you're the guy who's trailing, the correct decision is to call and not fold. But what you are really hoping for is for everybody to check, so you can see that last card for free.
Another comment re: success rate vs. EPA, it's kinda fascinating that coaches have intuitively figured out the minimax for coming up with short-term gains. In the emotion of the moment, it is very easy to get caught up in the short game and not the bigger game.
I think that's what's going on here--coaches are failing to think several steeps ahead. We see this a lot with the mismanagement of timeouts, coaches who throw on 3rd & 3 then go for it on 4th down, coaches who don't want to risk giving up 3 points on 4th & goal when even a failure could lead to possession of the ball & more points anyway.
Brian, I think the main reason you are having trouble finding a correlation between run success and pass success from a game theory perspective is because you are simplifying the game into two basic strategies (run/pass) when there are actually several substrategies that are as different from each other as "run" is from "pass", at least in terms of how they force the defense to react.
For example, the deep pass is used to set up the screen pass, the power run is used to set up the misdirection run, and so forth. A draw run has more in common with a screen pass than it does with a power iso run, so it is somewhat arbitrary to simply group all running plays together as one strategy and all passing plays as the other.
That being said, I realize that all of your analysis is derived from play-by-play data which only says "pass short left" or "run up the middle" with no indication of the actual play type, so I salute you for providing some of the best analysis of the available information. I just wish there was more information available because that could easily reduce the gulf between the x's and o's coaches and the stat geeks.
Also, you pretty much hit the nail on the head with this article. I agree wholeheartedly that coaches overvalue success rate (1st downs) and undervalue EPA/WPA (touchdowns). However, I would add that coaches are highly sensitive to expectations, and it is not entirely binomial (success/fail). For example, if a failure is expected, they will call a play that results in the least conspicuous type of failure, thereby exceeding expectations. That is exactly why coaches run the draw play on 3rd and long. They are basically saying, "Nobody expects us to get the 1st down, so let's just add seven yards to our punt with this draw play." It is the least conspicuous way to fail in that situation (although you still hear the boo-birds every time the home team does it in a close game, so it's not THAT inconspicuous). But if you're going to concede the punt on 3rd down, why not take a shot downfield instead? A 50-yard pass that is intercepted is no worse than a punt, and you will draw a PI flag or complete the pass every once in a while.
OT - poker discussion not really related to football.
"Not quite. By raising the pot, you give yourself an 80% chance at $130, as opposed to a 100% chance at $100. You increase your expected payoff by 4% if he calls."
Assuming you have the best hand after the turn, you are basically obligated into calling a river bet if the other guy hits his full house. In reality, not making a significant bet on the turn forces you into a really difficult situation on the river.
Even assuming there aren't going to be any more river bets, you are still off on the math. You'd take down a $130 pot 80% of the time, but $15 of that is your own turn bet. So the reality is that making a $15 turn bet leads to winning $115 80% of the time and losing $15 20% of the time. That adds up to only $89 of EV so you are better off taking down the $100 early on the turn.
When you throw in the possibility that somebody might call without pot odds and the savings you get from avoiding having to match a river bet, you really have considerably higher EV from forcing the issue at the turn. If you throw in the fact that he quite a few players call without pot odds and call again after missing the river, you're missing out an incredible amount of EV by keeping the turn bet small.
Its a huge mistake to ever give anyone else pot odds to make a call on a draw.
Brett,
You do realize that Brian did an analysis of run/pass WPA on different down/distances and found that teams should be running the ball a lot more on 3rd and long right? Teams should be running the draw more on 3rd and long, not less.
As former coach (albeit in the UK) and now a blogger following the NFL, with a healthy dislike of Statisticians, I'd like to offer a few thoughts to the debate which I hope will be taken in the correct context:
-- The reason I dislike/mistrust statisticians is because of the often smug sounding claims from the community of "we're right, all coaches are idiots". There are plenty of high school coaches on certain forums who have this attitude... and also have multiple losing records to their name.
-- When you hear a table of NFL commentators talk about the need "to run the football and stop the run", they might not all be talking about the same thing. Guy A might be talking about the red zone. Guy B is talking about 1st and 2nd down to put the defense in a 3rd and long (which is something, maybe from a success rate POV, someone should study). Guy C might just be talking about in the fourth quarter. Bare that thought in mind.
-- Disparity in teams, specifically;
a) There is a difference between Peyton Manning and Alex Smith
b) There is a difference between Reggie Wayne and Darious Heyward-Bey
c) There is a difference between the Colts O-line and the Bears O-line.
d) There is a difference between Adrian Peterson and Pierre Thomas.
e) There is a difference between the Steelers rushing D and the Bills rushing D
f) There is a difference between the Steelers pass D and the Texans pass D
g) All these are relative to the current season. Players come, players go. Coaches move on.
-- Ultimately all statisticians tend to view football with the "let's optimise our winning strategy" lens. Coaches, being paid millions of dollars tend to view it with a "let's keep it close at least" lens. Statisticians are trying to find optimal strategies for each situation. Coaches are trying to win but without losing their jobs in the process.
Brendan,
I think the reason teams appear to be more successful at running on 3rd and long is because of the type of pass plays usually being called. Most teams are calling screens or quick underneath routes simply trying to gain field position before the punt, which is essentially the same thing as a draw run. It goes back to my previous post about Brian's analysis not taking into account the specific types of pass and run plays being called. From a strategy standpoint, a screen pass should be in the same category as a draw run because they are both designed to counter the same defensive strategy. This is not a knock on Brian's analysis, as I agree that running more on 3rd down is generally a good idea, but not on 3rd and long unless the clock and score dictate it or if the team is planning on going for it on 4th down.
Also, I was only referring to 3rd-and-long situations outside FG range. Brian's analysis included plays inside FG range, and he reasoned that running in this situation results in a better EPA because it increases the likelihood of a FG due to the lower risk of a turnover or loss of yards. I agree with this reasoning. However, outside FG range a sack on 3rd-and-long is not nearly as detrimental to EPA, and as I stated earlier, a long interception is the same as a punt. The potential rewards of a deep pass in this situation significantly outweigh the risks.
Why did you base success rate on EPA rather than WPA? Just curious... I'd be interested to see that statistic (percentage of plays which resulted in a positive WPA).
"The correlation between offensive run and pass EPA is smaller than for SR—0.35."
Brian, I don't understand this statement"
are you comparing offensive run to pass EPA (which it sounds like, but I don't think you mean), or saying that the correlation between offensive run EPA and team wins and pass epa and team wins is smaller than for running success rate?
The correlation b/w run and pass EPA is less than that b/w run and pass SR.