## More SB 47 Analysis

I was just getting started with a more full analysis of the SB game decisions. I wanted to take a detailed look at BAL's two FG decisions and SF's decision not to go for 2 when down 28-13 (28-12 prior to the XP). Just when I finished doing the calculations for BAL's 4th and 2 from the SF 20 and their 4th and goal from the SF 1, I got an email from intrepid reader-correspondent Matt Meiselman.

Matt kindly sent his own analysis, using the 4th down and win probability calculators. This is exactly what those calculators are for...so I don't have to do all the work! Take it away, Matt:

In a Super Bowl with colossal storylines, including the impending retirement of Ray Lewis and a 34-minute blackout delay, the battle of the Harbaughs was arguably the biggest. Both coaches are considered among the league’s elite, and there won’t be much criticism of either of them following such a well-played season by both the Ravens and 49ers. In the Super Bowl, however, there were a few decisions made by the Harbaugh brothers on 4th down that significantly decreased their respective teams’ chances of winning.

There is no NFL head coach that will go for it on fourth down from inside their own 20 yard line in the first half of a game. Surprisingly, according to the 4th down calculator at advancednflstats.com, coaches are making a mistake by punting the ball away in many of these situations. With 2:07 left in the 2nd quarter, the Niners had a 4th and 4 from their own 12. Down just 14-3, there wasn’t too much desperation from the San Francisco coaching staff and they sent on the punt team without a second thought. In reality, going for it in this situation would not have been a desperate move. By punting the ball, the Niners actually decreased their chances of winning by 2 percent. Keeping the ball had tremendous value in this situation, and a punt would likely give the Ravens good field position and a decent chance at scoring anyway. That’s exactly what happened when Joe Flacco connected with Jacoby Jones for a touchdown just 22 seconds later.

Following the delay, the Niners stormed back into the game, cutting the lead to 28-20. They drove down the field, but eventually stalled at the Baltimore 16 yard line. 3:10 remained in the 3rd quarter, and the 49ers needed just 2 yards for a first down, but they settled for a David Akers field goal attempt. Forget the fact that Akers has been abysmal of late; this decision was an egregious mistake even with an ideal field goal kicker. A 4th and 2 is converted roughly 60 percent of the time, and that figure is likely higher with the Niners dominant rushing attack. If you merely assumed average circumstances, the Niners still decreased their odds of winning by about 4 percent by choosing to kick the field goal.

The biggest coaching blunder of this game was actually made by the winning coach, John Harbaugh, and it occurred with 4:19 remaining in the game. The Ravens held a 2-point lead, and were facing a 4th and 2 from the San Francisco 20 yard line. Conventional wisdom suggests that you kick a field goal to go up 5, forcing the other team to have to score a touchdown to beat you. However, the value of converting the first down and potentially going up by 9, which would essentially end the game, makes going for it on 4th down a no-brainer in this situation. On the field goal attempt, the Ravens had a 71 percent chance of winning, but if they had gone for it instead, that number would have increased to 78 percent. The Ravens gave up 7 percent of a win before the play even started.

This game also contained a monumental gaffe in terms of 2-point conversion decision-making. When Jim Harbaugh and the 49ers scored to cut the lead to 28-12 (pending the XP/2-pt attempt), it was an ideal time to attempt a 2-point conversion. The win probability calculations suggest that by going for 2, the Niners had a 5 percent chance of winning compared to 4 percent by kicking the extra point. This change is meaningful, but it doesn’t tell the whole story. If the Niners were to come back to tie from down 22, they would need 3 scores including one conversion. Coaches typically kick extra points in this scenario, delaying a 2-point attempt until the last score. But if San Francisco goes for 2 on the first score, they cut the deficit to 14, which means they don’t need any more conversions. So it doesn’t matter when they go for 2, right? Wrong. If the 49ers missed the 2-point attempt on the first touchdown, they still would have had roughly a 25% chance of converting back-to-back 2-point conversions to cut the lead from 16 to 8 to 0 (This assumes a 2-point attempt has about a 50 percent success rate, but with Kaepernick and the read-option, the odds were probably more in San Fran’s favor). This 25 percent increase is seen in the jump from a 4 percent win probability up to 5 percent.

In a total of 4 plays, the two coaches cost themselves 7 percent of a win each. This may not seem like much, but extrapolate that over a whole season and you get 112 percent of a win (16 x .07), or 1.12 wins, which is more than considerable. And remember, these are two of the best coaches in the NFL. Imagine how many wins the coaches at the bottom of the barrel are costing their teams.

(14-3) 4th and 4 for SF on own 12
2:07 2nd quarter
WP go for it: .12
WP punt: .10
Loss of .02

(28-20) 4th and 2 for SF on opp 16
3:10 3rd quarter
WP go for it: .30
WP fg attempt: .26
Loss of .04

(31-29) 4th and 2 for BAL on opp 20
4:19 4th quarter
WP go for it: .78
WP fg attempt: .71
Loss of .07

Niners TD (28-12) SF
WP go for it: .05
WP XP: .04
Loss of .01

### 33 Responses to “More SB 47 Analysis”

1. James says:

Sorry to be off topic, but the 2012 stats include the stats from the Pro Bowl. Either that or Eli Manning played a really good playoff game that I completely missed...

2. Brian Burke says:

It happened by accident last night when the SB stats got sucked into the system. I'm planning to get rid of them tonight. Thanks for the heads up.

3. Anonymous says:

Shove that calculator up ur ass. Balt goes on 4th down in 4th and doesn't get it, 9ers kick a fg a win game. Idiot

4. Anonymous says:

You're being a bit harsh, I'd say. But I agree with ur point. With the way Bmore's bend-don't-break (ie give up a TD) defense was playing, I'd have taken the FG there, even with a 75-80% conversion expectation (not sure what the calculator's assumption was).

5. Dale says:

I can't imagine how stats from a Pro Bowl would skew things...

6. Erik Jensen says:

Anonymous, the calculation takes into account the possibilities (not certainties) of a failed conversion and SF field goal. Every decision in football is taking a chance, even kicking the field goal. That choice leads to a lower probability of winning.

I'm sure you keep Vegas quite happy with your "damn the odds, go with your gut" philosophy. Thanks for keeping the lights on so the rest of us can enjoy a cheap buffet.

7. Anonymous says:

I will say that Ravens struggled late in short yardage. They only barely converted the 4th and inches with that Boldin catch. What odds would you give on that again? With Pierce banged up and Rice not looking great, there may be some situational factors pointing towards kicking the FG at least worth considering.

8. Anonymous says:

This is going to be out there so be prepared. Lets say the ref's called holding, like they should have, on the safety, moving the ball back 10 yards. What would be the difference chance that SF (Balts punter had previously miss-hit a punt in the game) get the ball with a reasonable (50-50) shot at a free kick. Maybe there isn't a good chance at all, not to mention Akers was imploding.. But to my eyes, without knowing any stats on it, it seems that the no call holding on the safety dramatically reduced SF's odds of tieing the game.

9. Jeff says:

Holding in the end zone results in a safety. The non-call doesn't matter at all.

10. Anonymous says:

I disagree that San Fran's read option increases their odds of converting a two-point conversion. Their offense allows them to gash the defense with big plays in the middle of the field. But in the read zone I'll take a precision passer instead. We saw how Kaepernick struggled with the defense packed into the box inside the 5 on their last drive. I still would have gone for 2 when it was 28-12. But you need to be careful building a statistical argument and then throwing in a very debatable point as the cherry in top.

11. Martin says:

What about the decision not to go for 2 on the first SF TD? Let's you got a 50/50 shot on each convertion. Then you have a 50/50 shot of not needing anymore. Throw in the fact that you then know that you need 2 points on both of the next TDs, 25% shot of making both, and then it's a much better option than waiting for the last TD

12. Jonathan says:

"Shove that calculator up ur ass."

That's the kid of insightful, quantitative analysis that I'm afraid this blog will never be able to match.

13. TPM says:

Matt -

This is great stuff, thanks for putting it together. I generally agree with you except for the BAL 4-and-2 on SF 20 with 4:19 to in the 4th. At the time, I was yelling at the TV "GO FOR IT! PUT THIS GAME AWAY! YOU'LL BE SORRY LATER!" But looking back on it, I can see a reason why they did it:
SF was playing pretty well at this point, and they seemed to have the momentum (I know momentum is vague and can't be measured, I read Brian B.'s great article on 5th Down, but just go with me on this). Although the Ravens' chances of making the 2 yards is high, if they failed, the morale shift, the psychological impact on both teams would have been HUGE. It's demoralizing for the offense and gives a second wind to the defense. I believe this is what John Harbaugh may have been thinking...he didn't want to give SF any more "mojo" than they already had.
Not saying it was necessarily the "correct" decision, but offering up a reason as to why Harbaugh did it, and why it wasn't necessarily the "wrong" decision.

14. Anonymous says:

no offense, but you fail high school statistics.

actually basing decisions on a 2% change in WP is ridiculous.

and claiming a 25% increase when a percentage changes from 4% to 5% is also ridiculous.

the correct conclusion would be to state that those decisions make no significant difference.

Articles like this give "advanced analysis" a black eye.

15. Steve says:

I think Jimmy Smith's hold/PI should get more attention. From what he said after the game it 's pretty clear the move was an intentional strategic move (as was the holding on the punt).

The basic idea is that, if Smith doesn't get flagged, they win. If he doesn't hold, it's a zero blitz so he's been hung out to dry. What are the odds he can defend Crabtree without holding/being hands on? Probably lower than the odds of not being flagged when you consider, as he said, how the game had been called.

On the previous play Smith made helmet to head contact on a defenseless receiver, no flag. Corey Graham shoved an official, no ejection... no flag. Torey Smith's obvious offensive PI wasn't called earlier in the game, and in general the 49ers and Ravens had gotten away with very physical '03 Pats/Ty Law style coverage all game with only one exception (the the big 3rd down PI against the 49ers).

When you look at the evidence I think you have to agree with Smith that the refs were unlikely to throw the flag pretty much no matter how much

16. CursoryComb says:

@Jeff. Holding in the endzone is a 10yard penalty enforced on the kick. 10 yards changes all the percentages when you're talking about a fair catch free kick. To say it doesn't is just like saying, "why look at any of the percentages." Holding should have been called on the safety and that is wasn't is appalling and shows that the refs deliberately were told not to call it. Akers was kicking deep into the endzone for most of the game from the standard kick off distance. What would be his percentage 10, 15, 20 yards closer? I couldn't care less about the PI, that isn't the issue. The no-call-holding safety is telling and could have obviously changed percentages for San Fran.

17. Brett says:

Holding in the endzone is a safety. Even if it had occurred outside the end zone and it was a 10-yard or half-distance penalty, they could have just taken the safety on the next play and run off even more time, so I don't understand all the uproar over that play. It was a smart decision by the Ravens because there is no way any penalty can hurt you when you are trying to take a safety. I would have loved to see the 49ers do a fair-catch free kick to end the game though, but I'd be surprised if Harbaugh even knows about that rule.

18. Anonymous says:

In response to the criticism about the 25% increase, it's wrong to say that those changes are insignificant. By that logic, if a team is down by several scores and has a .01 or .02 WP, all of their decisions are meaningless because they don't change the probability by that much (say it drops to 0%). These percentage changes are absolutely significant.

19. Steve says:

@Brett. The punishment for intentionally violating the rules to get an advantage is under the discretion of the refs. The rulebook only makes specific suggestions about what should be done in two cases, but a 10-20 yard penalty seems reasonable.

20. J.R. says:

@Steve, do you really think the refs are going to enforce the "palpably unfair act" in the waning seconds of the Super Bowl, when it has not been called in years? Especially when Rule 14, Section 1, Article 11, A.R. 14 gives the following example of a safety occuring simultaneously with a penalty-for-safety: "Second-and-15 on A4. Runner A1 is downed in the end zone. During the run A2 held on the A10. Ruling: Safety. Decline penalty."

I'm not 100% but I believe the last two words suggest that the penalty is declined by rule.

21. tgt says:

The "palpably unfair acts" tend to be those that violate the norms of the sport, like people coming off the bench. A player on the field holding an opposing player to keep the opposing player from tackling someone? Not outside the norms.

Yes, this was a spot where the rules could be gamed, but no more than horse collar tackles on a breakaway runner or holding a receiver that you know would beat you.

22. Jonathan says:

"I think Jimmy Smith's hold/PI should get more attention. From what he said after the game it's pretty clear the move was an intentional strategic move (as was the holding on the punt)."

Steve, this is a classic example of why officials should NOT "swallow their whistles" on important plays. Officials that decide to "avoid affecting the outcome of the game" end up doing exactly that--affecting the outcome of the game.

23. Jonathan says:

As for the holding, I don't really see the issue. If they called holding in the end zone, it's a safety anyway. If they call safety outside the end zone, its declined and it's a safety. If they decided to accept the penalty, it's not applied to the kickoff. It's just 4th down again.

I don't see the problem. You're not going to punish a team for taking an intentional safety, or for kicking an OT field goal on 1st & goal, or otherwise intentionally forfeiting points. Why punish a team for accepting a safety by holding in the end zone?

24. SlackerInc says:
This comment has been removed by the author.
25. SlackerInc says:

I want to talk about what happened after the free kick. It was a typically poor, halfhearted attempt to make a desperation series of laterals to try to win the game on the last play. Why do NFL teams so rarely seem to make a reasonable effort at a "Stanford band" type of play? Too little practice at it? Dearth of imagination? Afraid of looking foolish? Or: do they want the final score to go down in the record books as looking close, rather than risk the other team picking up a fumble and running it the other way for a garbage TD?

26. Greg says:

Anonymous if you deem the .01 or .02 WP to be meaningless than you may as well say the majority of the plays in a football game hold no value. Most plays do not have a huge effect but they do swing each teams probability of winning. The players have greater impact on the game than do the coaches, however the coaches still do a have an impact on the game. Every decision and play has a some impact on the game and it is all of these added together that determines the outcome.

27. FarHorizon says:

"and claiming a 25% increase when a percentage changes from 4% to 5% is also ridiculous."

Actually ... this shows you have no clue at all about how statistics work :)

28. SlackerInc says:

I will give the poster some benefit of the doubt and assume the poster was thinking there is a margin of error like in polling, where the difference between 4% and 5% is not significant, like the difference between 40 and 50 percent would be, even if the ratios are the same. I don't know if the analogy holds here, but given Nate Silver's history before 538, his refernces to this blog, and his ongoing use of analogies in the other direction, I've got to suspect there might be some validity in raising that objection, although it certainly could have been done more tactfully.

29. Steve says:

@J.R. The example you cited is for simultaneous events. You would think the intent of holding in order to burn the clock before taking the safety makes it clear the events weren't simultaneous.

I didn't expect the refs to do anything, it was the Super Bowl! The league ran 100 NFLEvolution ads during the game but the refs didn't flag Jimmy Smith for an obvious for helmet to helmet hit that caused a concussion. They don't eject, flag, or fine players for shoving refs, which probably violates the ref union contract. The Super Bowl is like the wild west, but when the regular season starts they'll go back to enforcing helmet to helmets, defensive holding, etc.

30. Anonymous says:

Brian, can you do an analysis on Harbaugh's decision to take a TO at the 5 yard line with 3rd and goal. Do the 9ers having a better chance at winning with 3rd and goal plus 2 timeouts in case they are stopped?

31. CursoryComb says:

Ok, might be a little late to the party for the Safety, but holding, in the endzone is a safety with the penalty being enforced on the kickoff. This is how i've always understood the rule. (A penalty on a play before a kick, can be enforced on the kick) Punting from the 10 yard line would seem to be much better odds for a fair catch free kick.

@jonathan You punish a team when they break the rules. That's the strangest argument i've ever heard. Yea he tore the guys balls off but they were taking an intentional safety why punish the team.

This isn't Nam. There are rules Smokey.