Romer 4th Down Study Underestimates Itself

Economist David Romer's 2005 study of when NFL offenses should go for it on 4th down rather than kick is possibly the most original, clever, and conclusive research of its kind. Romer's study concludes that offenses should go for the 1st down far more often than they typically do.

Beyond all the economist jibber-jabber in the paper, it is devilishly simple and compelling. Based on the concept of expected points (the average points scored from a first down at each yard line), Romer compares the average expected point value resulting from three options: punts, field goal attempts, and going for the 1st down. Ultimately, given the field position and to go distance, Romer recommends when to choose each option.

But I think his conclusions are wrong. Or perhaps I should say they're even more right than previously thought.

To explain the research itself and why I think it misses the mark, I'll illustrate an example. Say a team faces a 4th down and 4 from its opponent's 40 yard line in the first quarter.

A team would typically punt in that situation. On average, punting from the 40 results in an opponent having field position at its own 12 yard line. The opponent would have about -0.2 expected points in that situation. (+0.2 points for our team) That's the conventional decision.

Attempting a field goal wouldn't be wise. On average, a 57-yard FG attempt is only successful 36% of the time. You'd think the expected points of a made FG is obviously 3, but it isn't. It's actually 2.3 because the ensuing kickoff hands the ball to the opponent at, on average, their own 27. That situation has an expected point value of 0.7. (TDs are likewise worth only 6.3 points.) The other 64% of the time, the unsuccessful FG would give the ball to the opponent at their own 47, with an expected value of 2.1 points. The resulting total expected value of an attempted FG is:

0.36 * 2.3 + 0.64 * (-2.1) = -0.5 exp pts

So far, punting is the better option from the 40. But what about going for the 1st down? Attempting the 1st down conversion is risky, but the payoff may outweigh the risk. With 4 yards to go, the probability of conversion is 0.52. So 48% of the time, the offense would be unsuccessful and turn the ball over at (about) the 40. That would be 1.5 expected points for the opponents, or -1.5 points to our team. But 52% of the time we'd get the first down yielding 2.5 expected points. The total expected value of going for the 1st down is:

0.52 * 2.5 + 0.48 * (-1.5) = 0.6 exp pts.

All things considered, going for it is worth +0.6 pts, a FG attempt is worth -0.5 pts, and punting is worth +0.2 pts. We can conclude that going for the 1st down on 4th and 4 from the 40 is usually the right decision.

The only problem is that going for it really isn't worth +0.6 expected points. It's actually worth more. I think Romer underestimates the value of the "go for it" strategy.

Remember that the entire expected point curve is based on an empirical observation of actual NFL drives. As a rule, those drives were characterized by the "kick on anything longer than 4th and inches" strategy. If a team actually employed Romer's recommendations, their expected point curve would steepen, resulting in ever more favorable values for 1st downs. In my example above, a team shouldn't go for it on the 40 only to revert to conventional tactics at the 30 or 20 or 10 yard line. A "Romer offense's" expected point value at the 40 would not really be 2.5, but something even higher.

In other words, it's not a static process. The Romer strategy is a dynamic, recursive process that redefines the expected point curve itself. Brilliant as it is, the Romer paper underestimates its own implications. The paper's recommendations are a departure from the conservative style of football currently played in the NFL. And if I'm right, it would mean the optimum strategy should be even more radical.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

14 Responses to “Romer 4th Down Study Underestimates Itself”

  1. Anonymous says:

    Very clever.

  2. Anonymous says:

    You're right, but you're also opening a can of worms. Romer is not trying to define the proper strategy, but to show that coaches are afraid to deviate from the conventional wisdom. So if nothing else changes, going for it on 4th down tends to be a better strategy, also known as a partial equilibrium analysis. If coaches started going for it on 4th down, as per the recommendations, defenses might also change and that's your can of worms. For example, defenses might blitz even more on 3rd down in order to make the 4th down attempt tougher. It's next to impossible to extrapolate the partial equilibrium to a general equilibrium model.

    BTW, where did the numbers on the diagram come from?

  3. Brian Burke says:

    True. Offenses might run more often too knowing 4th down was available. It's not too hard to get 10 yds in 4 tries running the ball. Defenses would have to respond to that as well. The resulting equilibrium wouldn't resemble what we see today. It could be the Anti-CFL. Lots of running, very little punting, maybe less short passing and more bombs. Who knows?

    Re the graph: I traced an approximate actual expected point curve from 1st and 3rd quarters of all regular season games 2000-2008. The "Romer offense" curve is just an illustration.

  4. Anonymous says:

    You're assuming that the difference is one-sided. If the opposing team is also playing aggressively on 4th down, then the difference in field position will also be giving more points when the 4th down play fails, since the difference in expected points with an aggressive strategy would, I expect, be more pronounced starting on the 40 vs. the 12, since you can't really play that much more aggressively on 4th deep in your own end.

  5. Brian Burke says:

    The "aggressiveness" of the strategies is in the decision to kick or not. The defense doesn't get a vote.

    But you're right in that the defense could sell out on a blitz or something. But they would be making a big mistake if they did that.

    We can safely say that the offense's chance of converting on 3rd down is definitely *no worse* than they had on 3rd down and the same to go distance. All they need to do is choose the same type of plays they would have on 3rd down. If the defense does something different than they would normally would have on 3rd, that only helps the offense.

    Assuming the defense's strategy is rational on 3rd down, if they alter their strategy equilibrium on 4th down they would be choosing a sub-optimum decision. The result would be an even higher probability of conversion for the offense.

    In game theory, the 4th down decision is considered a "2-player zero-sum game with perfect information." It's perfect information because the defense knows whether or not the offense is going to kick or go for it.

  6. Anonymous says:

    unless of course it's a fake.

  7. Anonymous says:

    Actually, Romer already beat you to your point. I was reading the 2005 version (I had only read the older version) and I don't remember reading this before:

    Partial Equilibrium versus General Equilibrium.

    The analysis looks at decisions on individual fourth downs, taking all other decisions as given. But these decisions could affect other choices. If both teams follow the recommendations of the dynamic-programming analysis, their offenses will do better on average. This suggests that the value of having the ball anywhere on the field will be greater than the partial-equilibrium estimates imply. In this case, the benefits of going for it relative to kicking will be even larger than the preceding analysis suggests, particularly when teams are far from their opponents’ goal line. If only one team is more aggressive on fourth downs, on the other hand, it will on average score more points; this implies that any situation (including ones where its opponent has the ball) will be more valuable to it. There is no evident reason for this to have an important effect on optimal choices. It appears likely, however, that the increase in value will be larger in situations where the team is more likely to face a fourth down soon, such as when it has the ball in near its own goal line. If this is correct, it implies that the analysis tends to understate the value of going for it near one’s goal line.

  8. Anonymous says:

    I'd point out that the team to team variability on this analysis would be huge, and negate any "rule of thumb" to go for it.

    By definition of an average success rate of 52% for 4th and 4, half the teams are below that.(if i may assume a mean is a median, but you get my point)

    Case in point, on thanksgiving day, the lions should not go for it against the titans (the lions should probably punt on third down. haha.)

    On the other hand, i bet the titans could always go for 4th down against the lions.


  9. Brian Burke says:

    Steve-Thanks. I should have known!

    Bob-I don't think it would negate the rule of thumb, but it would definitely need to be tailored to a team's strengths and weaknesses.

    Regarding weak vs strong teams, the opposite may be true. It may be the weaker teams that need this strategy the most. The strategy would increase the variance of points scored and allowed, which is exactly what a weaker opponent wants. It might be more like baseball, with larger game-to-game variance, which is what allows the Royals to somewhat frequently beat the Red Sox.

  10. Anonymous says:

    "Regarding weak vs strong teams, the opposite may be true. It may be the weaker teams that need this strategy the most. "

    according to the rule of thumb, yes weaker teams need it the most - but that could be wrong.

    We can figure out the %success requ'd to make it a 'good gamble'.

    x * 2.5 - (1-x) * (1.5) = -0.5 exp pts.

    or x = 0.25%

    so if a team has a worse than 25% chance then they should kick. We can take this number, to see what the max distance for 4th down can be, in order to go4it and kick to give the same expected points.

  11. Brian Burke says:

    Romer does that in his paper. There's a nice graph at the back that shows recommended maximum to go distances on 4th downs.

    I agree with this analysis, except when there is a mismatch in team strength. At some point in a game, a weaker team that is trailing is going to need some luck to win. So even when the percentages say 'punt,' a weaker team should roll the dice and go for it if they want a realistic chance of winning.

  12. Anonymous says:

    "Assuming the defense's strategy is rational on 3rd down, if they alter their strategy equilibrium on 4th down they would be choosing a sub-optimum decision. The result would be an even higher probability of conversion for the offense."

    False. This is because the offense shouldn't run the same types of plays on 3rd & 4 as it would on 4th & 4 - for the very reasons set out here. The expected cost of failure of, say, a run play to get four yards isn't spectacularly high. If you get 3 1/2, you can go for it on 4th. Or punt. On 4th down, though, the cost to the offense of a gain of less than four is much higher, so they should pick a play with a higher expected gain, meaning that the optimal defensive strategy is likely different.

  13. Brian Burke says:

    Ok. I see your point, but I don't know if that's going to make any difference. I've never heard of a coach making 3rd down play calls so that he can just get closer for a 4th down attempt.

    Assuming the goal is the same--to get the first down--then nothing should change. If your strategy mix is exactly the same as it would be for 3rd down, you are guaranteed at least the same rate of success.

  14. Z-Dog says:

    @Brian Burke

    It's not that coaches call a play on 3rd down just to get them closer on 4th down. It's that they'll call plays that are higher-risk on 3rd down, knowing that they still have a 4th down to get the conversion. This is particularly true in 3rd down passing situations.

    Think of it this way: the probability of converting 3rd & 10 followed by 4th and 10 is less than the probability of converting 3rd & 10 followed by 4th & 2. So on 3rd & 10, you might call something (say, a draw play) that has a low chance (but not chance) of getting 10 yards, but has a very high chance of getting you 6-8 yards, rather than a play that has an ok chance of getting you 10+ yards (a pass play) but also a very good chance of getting you zero or negative yards.

Leave a Reply

Note: Only a member of this blog may post a comment.