Sterling Moore and Split-Play WPA

Sterling Moore posted a stellar +0.47 +WPA in Sunday's AFC championship game. That's very good -- only Patrick Willis (+0.52) and teammate Dane Fletcher (+0.49) beat that mark for defenders on championship weekend. After all, Moore made arguably the defensive play of the weekend when he knocked what would have been the go-ahead touchdown pass out of Lee Evans's hands with mere seconds to go.

But most of Moore's +WPA actually comes from his contributions on the following play, the failed third down pass targeted for Dennis Pitta which set up the fateful fourth down on which Billy Cundiff kicked the Ravens out of the playoffs. The Ravens were still in excellent shape on that third down, and the failure to convert or score a touchdown took their win probability down from 83% to 43%, giving Moore a +0.40 WPA on the play. That leaves just +0.07 for his other successful play, the strip of Evans in the end zone.

That seems intuitively way too low, and that intuition is correct. Although technically the entire play from snap to throw to almost-catch to strip just cost the Ravens 7% of a win, if Evans holds on to the ball and Moore doesn't strip it, Baltimore's ticket to the Super Bowl is all but punched. But with the way the data is fed into our system, it's impossible to give out separate credit for different aspects of plays.

But let's say for a second it was possible. How would each aspect of that play have played out in the eyes of WPA?

Snap: Second and 1 from the NE 14, 27 seconds remain: 90% win probability.

Things looked great for the Ravens even before this play. With just 14 yards needed for a touchdown and the field goal if they couldn't punch it in just a mere chip shot, the Ravens were sitting pretty. Basically, the only two things which could sink the Ravens were a turnover or the missed field goal.

Throw and catch: Ravens lead 26-23 pending extra point, 22 seconds left: 99% win probability.

It's not over until 0:00, but the Patriots would have needed a Music City Miracle-esque finish to take this one back had Evans held on to the pass for the touchdown. Moore was beat by Evans on the play, but obviously not by much, because...

Strip: Third and 1 from the NE 14, 22 seconds remain: 83% win probability.

Although we can debate how Moore should be credited for this play -- was the throw good enough we don't blame him for less-than-perfect coverage and we merely credit him for the strip, or does the strip just make up for allowing the ball to get to Evans in the first place? Regardless, the actual act of the strip brought the Patriots win probability up from 1% to 17%

So how would I give credit for this play?

Flacco, Evans: +0.09 WPA for the pass and "catch."
Evans: -0.16 WPA for the drop, for a total of -0.07.
Moore: +0.16 +WPA for the strip.

The question of credit is one of the biggest questions those who use and develop statistics for team sports need to answer. Unfortunately, trying to answer these questions in real time makes it almost impossible to give fair answers, and given the subjectivity of certain things, it's almost impossible in retrospect as well.

Keep in mind with the version of WPA presented here, we are only able to give credit based on the entirety of the play. If you see a number that doesn't match up with intuition or common sense, remember the inner workings of an entire play may hold the answer.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

18 Responses to “Sterling Moore and Split-Play WPA”

  1. Jared Doom says:

    "Third and 1 from the NE 14, 22 seconds remain: 83% win probability."

    That seems high. I'm assuming eventual TD leads to GWP ~ 1.0, eventual FG leads to GWP ~ 0.5, and failure to score leads to GWP ~ 0.

    Assuming the probability of FG OR TD is 0.95, that implies the probability of an eventual TD is 0.71 [(0.71 * 1.0 + (0.95 - 0.71) * 0.5 + (1.0 - 0.95) * 0) = 0.83].

    Given only 22 seconds left a 71% chance of scoring an eventual TD on 3rd & 1 from the opponent's 14 seems high. Am I wrong (assuming yes, but still surprised)?

  2. Anonymous says:

    Flacco screwed up that play. He clearly had enough room to run it and dive for the first down. Lots of room! Ugh.

  3. Anonymous says:

    I was also surprised with that, Jarod. I won't use your math (because I don't know it), but the probability of Baltimore kicking a FG at that point should have over 75%. Then it's overtime on the road against a superior team (per EPA).

    How that translates into the Ravens having a 83% WP, I don't know.

  4. Anonymous says:

    In the 2011 regular season, FGs of 1-29 yards were missed 11 times; total is 301 for 312, or 96% if I counted right. (I didn't count XPs, but there were 6 missed). So that's a good SWAG.

    Where the 83% comes from is that in addition to scoring the immediate third down touchdown, they could get a first down instead and run a couple more plays before kicking the FG. I don't believe the WP generator takes TOs into effect, and I can't remember if BAL had one left or not.

  5. Jared Doom says:

    Anon #3 - To be fair I did say *eventual* TD, so I was accounting for a TD on any following play, and still thought 83% was high, given the limited number of plays you can run with 22 seconds left.

    I think the WP model implicitly (kind of) accounts for timeouts because I think it is based on historical data (albeit probably smoothed for situations with sparse data points).

  6. Kulko says:

    Wo doesnt Know how many timeouts a Team has. It just considers how many Teams historically habe scored a Td, or Made a FG in that Situation. Teams typically pass in this Situation and that means 22 Sec correspond to about 4 plays. Even Ehen you waste a play in getting the 1st thats enough chances for scoring

  7. Independent George says:

    I might be misunderstanding the math, but I came up with about a 35% chance of a touchdown.

    Assuming a TD is ~100% WP, and a FG is ~100% overtime, the WP is the probability of a TD + probability of a FG + OT win

    0.83 = P(TD) + P(FG)*0.5

    If P(FG) = 0.95, then

    0.83 = P(TD) + 0.95*0.5
    0.83 = P(TD) + 0.475

    0.355 = P(TD)

  8. Anonymous says:

    Should say AFC, not NFC, in the firs sentence.

  9. Dave Archibald says:

    I'm not sure even with the further breakdown this intuitively makes sense. Moore's (first) play is significant because without it the Patriots lose - going from 90% probability of losing to essentially 100% is a more important play than going from 50% to 60% at some other point in the game.

    There's almost a compounding effect here: without Moore's 2nd-down play, the 3rd-down pass breakup and the FG miss (both more important in terms of WP) can't happen. Then again, without those two events, his pass breakup is just a footnote. So maybe WP is "right" and it just feels inaccurate because I have the benefit of hindsight.

  10. Jared Doom says:

    Independent George - FG & TD are mutually exclusive outcomes so P(FG) + P(TD) must be less than 1.

    In your example P(FG) + P(TD)= 1.305.

    In my example I assumed 5% chance of no score, which means 95% chance of score, so I assumed P(FG) + P(TD) = 0.95. If you assume TD gives 100% WP, FG gives 50% WP, and no score gives 0% WP, then you have
    0.83 = 100%*P(TD)+50%*P(FG)+0%*P(no score).

    I forget what this is called in linear algebra but it has a unique solution.

  11. Independent George says:

    Ah, crap, you're right.

  12. James says:

    There's something screwy with the Win Probability Calculator. It says that 3rd and 1 has a WP of 83%, while first and 10 from the same spot has a WP of 63%. I don't see how having a first down lowers your WP at all, much less so drastically. Even 1st and goal from the 8 only has a WP of 70%.

    Parameters: Score differential of -3, time left 0:22, 4th quarter, Field Position is opponent's 14, 3rd down and 1 to get WP of 83%.

    Changed field position to 13, down to 1 and "To Go" to 10, get WP of 63%.

  13. Dave Archibald says:
    This comment has been removed by the author.
  14. tunesmith says:

    I have seen some oddities with WPA in other situations. For instance, in New England's game against Denver in the Divisional round, New England had the ball on first down. They threw an incomplete pass, and their WPA went up, +0.08 . Brian, I have an open question about it on its "live" page if you want to check it out. Also, I thought I'd point out that the game logs that drive the graphs often have missing plays, almost always when the time-remaining is the same as a previous play. (I only notice this because I've absolutely loved poring through the data this season - thanks again for providing such a cool way of looking at everything!)

    By the way, I've seen a few other examples of split-play that could more clearly be identified. One clear example would be when a play gets called back due to penalty. Denver@Buffalo, week 16 - Eddie Royal returns a kickoff for a touchdown, nullified by penalty. The penalty was worth negative WPA, but not a lot. However, figuring in the positive WPA of the lost touchdown, it was a huge play. Although I'm not sure if you allocate WPA to players for penalties.

  15. Dave Archibald says:

    The EP aren't adjusted for time. With :27 seconds left on the 14, the Ravens are given a +4.74 EPA, with a 83% chance of a FD and a 45% chance of a TD. If I change that to 3 seconds left, the EP doesn't change, even though no coach in the league would (or should) go for a TD at that point.

    For WP to be accurate in end-of-game situations, it needs to take into account the clock as a factor. The Ravens didn't have a 45% chance of scoring a TD from the 14 with that little time left, because the number of plays they could run in 27 seconds with one time out (and needing to kick a FG if they didn't get the TD) was just as much a constraint as the downs were.

    EDIT: deleted some comments with typos.

  16. Dan S says:

    I think it's been well established in the past that WP values are not at all precise, particularly in the end-game situations that we're most interested in. And I don't mean that in an intuitive, "no way is that the actual prob" way. The calculator can clearly be shown to be directly contradictory of itself.

    Rather than try to smooth out the sparse and incomplete play-by-play data, it's probably better to write an algorithm that can calculate probabilities at the point where it because reasonable to predict the number of plays left in the game.

    In any case, I love this site but cringe any time I see a final drive WPA analysis like this one.

  17. Brian Burke says:

    Dan S makes a good point. The WP calculator puts out smoothed empirical averages. In very tight final-minute situations, it is imprecise.

    Often, when I'll do a specific analysis, I'll go back to the data and built a customized look at the situation to include timeouts and other factors. But not everyone has that luxury and it takes a lot of time.

    Other types of models may be superior at this point in such a game. It's one reason why I'm a fan of Keith's drive Markov model. For example, in many cases where the 2 models produce contradictions we can learn where teams are making sub-optimum choices.

  18. Anonymous says:

    i think this is stupid because you cant drop a catch like that with couple seconds to go there is no reason to drop that ball. The defender did what he had to do which is break that play up.

Leave a Reply

Note: Only a member of this blog may post a comment.