tag:blogger.com,1999:blog-38600807.post7292664772551719260..comments2023-11-05T04:16:44.937-05:00Comments on Advanced Football Analytics (formerly Advanced NFL Stats): Examining the Value of Coaches' ChallengesUnknownnoreply@blogger.comBlogger13125tag:blogger.com,1999:blog-38600807.post-40635389550093404062013-10-11T16:47:20.680-04:002013-10-11T16:47:20.680-04:00"And the final point was that challenges have..."And the final point was that challenges have decreased since the league began automatically reviewing all scoring plays and turnovers. Those are the highest leverage plays you can have, so the only plays coaches really need to challenge anymore are catch/no catch on a 1st down conversion, and the spot of the ball."<br /><br />This is partially, but not completely true. Almost-scores and almost-turnovers are not automatically reviewed, but have just as much leverage as scores and turnovers. Those still constitute a significant portion of challenges.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-49134146590138764832013-10-11T16:17:52.481-04:002013-10-11T16:17:52.481-04:00>I like the Hail Mary challenge explanation way...>I like the Hail Mary challenge explanation way more than my "Officials are biased" explanation - makes much more sense from a decision theory perspective.<br /><br />Agreed. Near the end of a game, when leverage is likely to be the highest, coaches are more willing to throw a challenge just in case, even if they know they have a 1% chance of success.<br /><br />Another thing I just realized - if the challenge is successful, the team gets their timeout back. If the team was planning on calling a timeout anyway, why NOT throw the flag? It's a free timeout in that case. Wait for a play with even a small chance of being overturned, even if it's not high leverage, and throw the flag. Challenges go away once you reach the 2 minute warning as well, so you may as well use them before then.<br /><br />And the final point was that challenges have decreased since the league began automatically reviewing all scoring plays and turnovers. Those are the highest leverage plays you can have, so the only plays coaches really need to challenge anymore are catch/no catch on a 1st down conversion, and the spot of the ball.<br /><br />The number of challenges has decreased sharply, from 249 in 2010, to 209 in 2011, to just 157 in 2012, for what that's worth.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-25682528965506045792013-10-11T10:05:55.659-04:002013-10-11T10:05:55.659-04:00I like the Hail Mary challenge explanation way mor...I like the Hail Mary challenge explanation way more than my "Officials are biased" explanation - makes much more sense from a decision theory perspective.KMeershttps://www.blogger.com/profile/13582870758936595449noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-19403306901725279152013-10-11T02:58:59.816-04:002013-10-11T02:58:59.816-04:00I think your analysis of the expected value of the...I think your analysis of the expected value of the average timeout is incorrect.<br /><br />If I understand correctly, you argue that because the coach chose to challenge rather than take a timeout, you can conclude<br /><br /> E(challenge) >= E(timeout)<br /><br />But you've only established this in situations *when the coach chose to challenge*. There's no reason to think those situations are good times to call a timeout. If timeouts are actually taken in better situations, then the average value of a timeout *when timeouts are actually taken* will be higher.Anonymoushttps://www.blogger.com/profile/00224072347996971132noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-72786899372994226832013-10-10T22:06:57.915-04:002013-10-10T22:06:57.915-04:00Interesting topic for consideration. I agree that...Interesting topic for consideration. I agree that there is certainly more about this that can be fleshed out and that there is huge room for coaches to improve here and we just don't understand the nuances of that as well as for fourth downs and passing v running.<br /><br />Is it possible that low leverage challenges in games that have a 14+ lead (you note that this accounts for half of the really low leverage red flags) are not irrational. In these games, challenges don't need anywhere near the expected payoff to be justified as they would in closer games: this seems the case because timeouts are certainly less valuable in blowout games than they are in close ones. As the data set becomes more robust it might be advisable to ignore those games or separate them into bins. You might see interesting results and be able to determine the value of a timeout in games of different score spreads too, if indeed your hypothesis about coaches rationally judging challenges and timeouts in terms of WPA (it seems to make sense to me).<br /><br />I'm also inclined to agree with those who are suspicious of the "officials are not likely to overrule high leverage plays" claim. If coaches are rational then they see that the "pot odds" favor them in high leverage situations even when the risk of success is low, as others have stated. Additionally, there is such intensive scrutiny on officials by both the public (via the media) and the league itself (internal grading) that I can't imagine this being a major factor. They have lots of camera angles and it seems like they get it right a really high percentage of the time to me--most errors are the officials applying a badly written rule correctly (think tuck rule or Calvin Johnson "catch" in Chicago).<br /><br />I was surprised that there wasn't an even greater concentration of failed challenges in this "high payoff long-shot" category.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-29001882329809067492013-10-10T17:38:41.723-04:002013-10-10T17:38:41.723-04:00I am not surprised to see a higher failure rate am...I am not surprised to see a higher failure rate among high-leverage challenges. My anecdotal observation is that coaches sometimes treat their challenge as a Hail Mary attempt. I have seen plenty of times when the replay seems to CLEARLY confirm the original call, and yet a coach challenges the call because A) a reversal would lead to a huge swing in leverage, and B) the coach is in a desperate situation and seemingly has little choice but to make such a specious challenge.bytebodgerhttps://www.blogger.com/profile/15989876051555196561noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-8466044278482578232013-10-10T17:26:50.967-04:002013-10-10T17:26:50.967-04:00I think we should expect to see a lot more failed ...I think we should expect to see a lot more failed challenges in the high-leverage range than we do. The higher the leverage, the lower the probability of reversal required for the challenge to have positive expected value.<br /><br />The unsuccessful challenges do appear to be more densely distributed in the higher leverage range, but not much. It should (ideally) be proportional, where double the leverage means the challenge should be half as successful.Brian Burkehttps://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-86093871810391020492013-10-10T17:02:47.032-04:002013-10-10T17:02:47.032-04:00There's some selection bias in the success or ...There's some selection bias in the success or failure of high/low leverage situations since it's much more likely a coach needs to be sure of a low leverage challenge than a high one. If you binned the leverage to get a success rate, what does the leverage*p(success) look like? I'd expect a flattish line, though there's probably not enough data above 0.2 leverage.meep_42https://www.blogger.com/profile/16551118003171851297noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-34961292624168449692013-10-10T16:24:35.569-04:002013-10-10T16:24:35.569-04:00"coaches (if they are behaving rationally) va..."coaches (if they are behaving rationally) value their timeouts, on average, at 0.03 WP at most." <br /><br />That is quite a statement. It is based on equation,<br /><br />"E(Challenge) ≥ E(Timeout) ≥ 0"<br /><br />But perhaps consider, strangely enough, that coaches values timeouts more than challenges. Challenges cannot be made at last 2 mins where they often have the most impact. And coaches cannot challenge at any point in a game (effectively challenge at least). Where as a time out can be called at any time.<br /><br />Maybe this proves that coaches value timeouts more than .03 WP.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-12567527238925587322013-10-10T15:11:41.331-04:002013-10-10T15:11:41.331-04:00> .... it seems at least possible that official...> .... it seems at least possible that officials are<br />> less likely to overturn plays that are hugely<br />> influential on the outcome of the game.<br /><br />I think coaches are also more likely to make long-shot challenges in high leverage situations.<br /><br />> ...the expected value of a challenge, which <br />> is just the challenge’s leverage multiplied the<br />> probability of success:<br /><br />That's an implicit assumption that the value of a failed challenge is zero, but the challenge will still - for example - stop the clock. (Under specific circumstances it can even stop the clock for a longer time than a time out would.)NateTGnoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-67891889933150135652013-10-10T15:02:53.883-04:002013-10-10T15:02:53.883-04:00"This plot shows that very low leverage chall..."This plot shows that very low leverage challenges succeeded more often than higher leverage challenges. In fact, after leverage reaches about 0.15 WP, challenges became much more likely to fail, and it seems at least possible that officials are less likely to overturn plays that are hugely influential on the outcome of the game."<br /><br />Couldn't this just be selection bias? A coach is more likely to challenge an "obvious" bad call even if it is low leverage. And a coach is more likely to challenge just an "iffy" call as long as the payoff is big enough from a WP standpoint.Mikenoreply@blogger.comtag:blogger.com,1999:blog-38600807.post-36118022318769029632013-10-10T14:37:05.391-04:002013-10-10T14:37:05.391-04:00From a bad reasoning standpoint could you re-run t...From a bad reasoning standpoint could you re-run this analysis changing one key factor, assume all games are tied, or no team is down by more than a touchdown (maybe 10 points).<br /><br />I only ask because part of coaching is also appeasing your players and fans. So while your analysis says what is the (real) value of a challenge I would be curious if this could be shifted to what is the fan (or player) perceived value (lets assume fans always believe there is a chance to win). This would see if coaches tend to challenge plays that in a close game have high win percentage value even though they don't in the actual event. Therefore (I think) if the hypothesis is true then similar types of plays are challenged no matter the game state. Potentially showing coaches don't adjust, or that they go through the motions because it is expected (good coaching or not).<br /><br />Another way to look at the value might be instead of raw WP change you could do % change in WP, so with a .5% WP a .1% change (from .5 to .6) is a 20% gain while with a 50% WP a .1% change is a .2% change. Do coaches challenges do a better job of maximizing WP% change instead of WP.Anonymoushttps://www.blogger.com/profile/06914168967401672142noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-25053923144370626232013-10-10T14:14:07.068-04:002013-10-10T14:14:07.068-04:00Related to your last statement, I was thinking one...Related to your last statement, I was thinking one of the difficulties in your analysis has to be what type of plays that were eligible to be challenged were not? As they were not challenged, we can't assess whether the challenge would be successful. However, while this would be a vast undertaking, the following issues would surely arise:<br />-Was the coach simply out of challenges, though he would like to challenge the play?<br />-Was the benefit of a successful challenge not worth the cost of using a challenge opportunity (e.g. a 10-yard gain on first down in the first quarter that the official spots just shy of the first down marker)?<br />-Was the likelihood of overturning the call on the field too low for the benefit of overturning the challenge?<br />-Would the play have been challenged but the offense ran the next play before the challenge flag could get thrown?MFLoGrassohttps://www.blogger.com/profile/16421311347964163796noreply@blogger.com