Belichick Cheating Evidence?

One of the more apparent signs that a spy or corrupt official is cheating is that he is living a lifestyle beyond his means.

If Belichick's Patriots exploited unfair advantages in stealing signs from opposing sidelines we would expect to see some sort of evidence that they won games "beyond their means." By means I am referring to the Patriots' passing and running performance on offense and defense.

By successfully exploiting stolen signs, we might expect the Patriots to choose to use that advantage on critical plays--3rd downs in the 4th quarter for example. These critical plays would heavily "leverage" performance on the field to be converted into wins. In other words, the Patriots would win more games than their on field stats would indicate.

This is exactly what we see in the data. Year-in and year-out, Belichick's Patriots have won about 2 more games than expected given their offensive and defensive efficiencies, including turnovers and penalties. No other modern team has even come close to the Patriots in consistently winning more games than their stats indicate. Could those extra wins be due to cheating?


For a comparison of other teams' actual/expected wins charts, see this article. When I first discovered this pattern, I believed it was evidence of Belichick's in-game "genius." He was known for some unconventional tactics, such as going for it on 4th downs more often than his counterparts. (I'd go for it on 4th down too if I knew the play the defense would run.)

In a previous article, I ranked every head coach since 1983 in terms of how many excess wins they had above their expected wins based on team efficiency. Belichick's career, as a whole, was rather average due to his poor results in Cleveland. But by isolating his tenure with New England, his excess wins per season would rank him as the best tactical coach ever, with an extra +2.33 wins per season. The rest of the pack is far behind with the 2nd best coach at +1.83 wins per season. Belichick is a true outlier, at almost 3 standard deviations above the mean.

This is only circumstantial evidence of cheating, but it is evidence. And although hardly damning, we can be sure of one thing about Belichick--he is willing to cheat. If someone has crossed the line by breaking one rule, what makes you think he's not willing to break others?

On some other sites I've read some emotional comments rationalizing the Patriots' cheating. Whether other teams have attempted this or not is not relevant. Patriots fans and Belichick fans must accept the possibility that much of their success has been due to cheating. What I've done here has added weight to that possibility and quantified the potential scope of this scandal. At the very least, we would be justified in continuing to investigate the Patriots' methods.

Note: Some other examples of teams' expected vs. actual wins can be found here. Also, since this post is appearing on many other sites recently, I've posted a response to many of the excellent comments and criticisms over the past several months.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

58 Responses to “Belichick Cheating Evidence?”

  1. Anonymous says:

    I understand that you're not offering this as any sort of definitive evidence, but for argument's sake...

    If it's true that the best teams tend to out preform their stats due to the short season; and if it's true that Bellichick's Patriots are one of the best teams, then this result isn't unreasonable.

    But this assumes that the Pats are great independent of any cheating.

    Have you (or anyone else) looked into the reasons why teams out preform their stats (be it just luck or some controllable variable)? I wonder how possible it is to isolate any evidence of cheating. My guess is not very.

    Not sure that makes any sense or is even interesting.

    Regardless, I agree with your conclusion.

  2. Brian Burke says:

    Pris-I share the same interpretation. It could be due to greatness, cheating, or luck--or a combination of any or all of the above.

    But, Belichick's score is SO high, at an amazing 3 standard deviations above the mean, and the result is SO consistent, it's hard to believe that just luck and skill are at play, given the fact we know he's willing to cheat.

    It's possible cheating had no effect, but why do it when your team is the odds-on favorite to win the Super Bowl and you're the world's best coach?

  3. Anonymous says:

    VERY - interesting!!! And I've forwarded this link to all my outspoken Patriot's fan friends. A thought that came to mind is that this analysis doesn't incorporate turnovers in as much as a team with a high positive turnover ratio might have a shorter field to score and their points/yds ratio would be higher while their opponent's points/yards ratio would be lower since many drives would have ended with a turnover.

    Of course, one way to increase your turnover ratio is to know what the other team is going to do so you would need to find a way to normalize for this which may be impossible.

  4. Brian Burke says:

    tb-It would take a month to find all the turnover data going back to 1983, but it would help. But for coaches with longer careers, the turnover factor will tend to even out.

    But, I do use all turnover data for recent years. So the graph in this post that illustrates how Belichick's Patriots have overperformed their stats by 2 wins per year does include turnover data.

    I don't have time to post all the other team's graphs (with turnover data). But trust me, no other team has a profile anything like the Patriot's. (Except maybe the Lions, who tend to lose more games than they should.)

  5. Anonymous says:

    Embarrassingly bad piece. Since they were stealing defensive signals, then that would benefit their offensive numbers across the board.

    So why wouldn't this show up in your stats?

    Feeble.

  6. Brian Burke says:

    To the anonymous Pats fan-

    1. The article does not assume that the Patriots rule-breaking activities were limited just to videotaping opponent's signals, and so the advantage would not be limited to improving offensive stats.

    2. The article points out that the Patriots would likely use their knowledge of opponent's defensive calls in high-leverage situations--4th and short, for example. So if It's 4th and 2, and the opponent calls a run blitz, the Pats could call a WR screen. If the play picks up 4 yds the Pats get a first down, but their average yds per pass would actually suffer.

    Please read the entire article before throwing out words like embarrassing and feeble.

    I'm curious how you would explain the Patriot's over-performance? The possibilities are:

    a. They are the luckiest sports team in the history of all sports.
    b. Belichick is not a cheater, but a football genius the likes of which come along once in 10,000 years.
    c. They cheated sometimes.

    We know one of the above is true already.

  7. Anonymous says:

    Since most of your "statistics" here are based on an assumption of cheating then why not assume that all other teams cheat in the same way, only not from on-field cameras. That's the only difference, an on-field camera versus one from the stands or box. Therefore, if all teams steal signals then your assertion of Belichik's over achieving is negated. What teams don't "scout" their opponents? When coaches look at films, what do you think are they looking at - vacation movies? When coaches speak behind their play sheets is it because they're all afraid THE BILL will be watching? Seriously, all coaches fear Belichik will be watching? Even the ones who will never play against NE? You're taking one exposed instance of what happens in every NFL game by every NFL team and blowing it into a big balloon filled with hot air. New England wins because the players are better prepared to play 60 minutes every game, especially in a very complex defense. They also have the best QB/receiver(s) combination in the league. That's why they win, not because they cheat.

    Why not just report and work with fact as they are instead of employing such useless speculation. If you're going to factor in drivel why not use blown calls by the officiating crews? How many of those have gone against NE compared to other teams? Do you have those stats? Everyone who has eyes can see those and it's not just left to speculation; so, they should be given more weight in your various equations than a supposition of spying. I doubt if you'd take that route – it's not in the interest of your hypothesis that NE just can't be good but must win by cheating. In the game of unbeaten teams, the Colts ran well in the first half and then were stopped in the second. The Patriots must have cheated better in the second half accoding to your theory. Phantom interference calls on both offense and defense against NE kept the Colts in the game, not the Colts outstanding play. Manning was shut down and then Addai was shut down. Brady again proved he's the best late-drive QB in the game. Is Belichik a better coach than Dungy? Yes, in this game, because the Colts had no answer for what NE was doing (Brady) and couldn't make second half adjustments and close the game. But you would put it down to cheating.

    Stick to real facts and numbers and leave the speculation to the idiots on the pre-game shows. They get paid to be stupid.

  8. Brian Burke says:

    You're obviously a very loyal Pats fan and want to defend your team. I get that. But no speculation is needed to know that Belichick cheated. It's just a matter of knowing how much and measuring the effect, right?

    The only speculation here is when people suggest everyone else cheats too.

    All I did here is show that NE consistently wins more games than we'd expect given their efficiency stats. No other team is remotely close to that record over the past several years. It's just evidence, not proof.

  9. Anonymous says:

    A simple test of your cheating hypothesis is to repeat the analysis in another couple years and see if there is a statistically significant decrease in their relative performance in that late-game, high-leverage situations you've focused on.

    If cheating was responsible for their over-production, then you'll see a significant decrease. If it was coach/team greatness then you won't.

    Of course, this is nullified if something traumatic happens like a major Brady injury, or something else that seriously confounds the study.

  10. Anonymous says:

    If the cheating has a positive effect for the Patriots, it affects their play on the field...so wouldn't the positive result of the cheating already be factored into their efficiency stats?

  11. Brian Burke says:

    Only if they're using the advantage most of the time. My guess is that they're reading defensive signals (and possibly others) and using them mostly on very high-leverage plays--plays that don't have a large impact on their stats but have a very large impact on the outcome of the game. If it's 3rd and 5 and they get a 6 yd pass, that doesn't help their passing stats much, but it keeps the winning drive alive in the 4th quarter.

    That would explain the "excess" wins above what would be expected given their on-field stats.

  12. Anonymous says:

    The only speculation here is when people suggest everyone else cheats too.

    My guess is that they're reading defensive signals (and possibly others) and using them mostly on very high-leverage plays.

    Look at the first synonym in this dictionary entry.

    This article is nothing but speculation. That's fine, but you should admit it. I mean there is definitely no evidence given for the "possibly others" remark.

    In your previous post, the Lions we far greater outliers than the Pats most seasons. Did they reverse cheat? Are they the unluckiest team ever?

  13. Anonymous says:

    I think the part that is specualtive is the assumption that the information gleaned from the cheating was only used at certain times--enough to affect wins and losses but not enough to affect efficienct stats, thereby nicely fitting this thesis. Maybe this is how it went down, but at this point it's definitely speculation.

  14. Brian Burke says:

    Doug-I think that's a fair point. I might characterize it as a hypothesis supported by at least some circumstantial evidence, and not as pure speculation. My use of the word "guess" wasn't wise. And Vaclav, that's incorrect about the Lions. The Pats are more consistent outliers, but I do think the Lions may be poorly run from top to bottom.

    My response about "the only speculation here" was related only to the previous rant from the Pats fan going on about unfair refs and the "every other team does it" argument.

    I don't mean to be lecturing people here, but that's how science and research work. People "speculate"--have theories--then test them with observation. If the observations support the theory, that's one more reason to believe it.

    Here's how I came to the theory:

    1. The Patriots did cheat. They were caught, warned, but continued. It's difficult, time consuming, and risky. From that, we'd be safe saying the filming had a purpose and a value. The Patriots themselves must have believed the practice gave them some kind of advantage.

    2. We know the Patriots' performance in 2nd halves were not significantly better than in 1st halves (see the related FO article). And we know that their performance against teams they faced twice in a season was not any better in second games than first games. It follows that they probably didn't systematically use an unfair advantage *all the time*.

    3. Accordingly, if they did gain an advantage (which they themselves believed), perhaps it was only applied in certain critical situations. The alternate (null) hypothesis is that they never gained an advantage--the Pats did all that risky work that was ultimately fruitless.

    I then remembered my look at "gameday" coaches and how Belichick's W-L record was off-the-charts above and beyond what you'd expect given his teams' on-field performance. I put 2 and 2 together and the numbers appear to support the theory.

    Thanks for keeping me honest.

  15. PantsB says:

    This still makes major assumptions.

    1- The equal applicability of your statistical methods across every team. If an agent acts in such a way that the system does not accurately predict outcomes, its not logical to assume the system is correct and the agent is breaking assumed rule sets in reaching outcomes outside your predictions. Since you seem to have a yardage based efficiency system, the most obvious point is the Patriot's tendency to have a bend-but-don't-break system. The inability of your projection system to account for this is more an indictment of your system than the agent that is outside your norm. When the Colt's broke the DVOA stats at FO, they didn't assume cheating, they reworked their system.
    Let's look at the Football Outsider's win projections/Pythagorean wins.

    Year Estimated Wins Actual Wins
    2006 11.0/12.2 12
    2005 8.9/9.1 10
    2004 13.1/12.4 14
    2003 11.6/11.4 14
    2002 9.3/8.9 9

    Only in 03 and 04 is there a variance of greater than 1 game, and any system that estimates wins is going to tend to have a greater error rate, usually towards the mean, with the great teams.

  16. Brian Burke says:

    Jake-Great comments. I agree with you on one point and disagree on the other.

    First, I agree that we can't just say any residuals/errors not accounted for by a model can automatically be chalked up to cheating. My original theory was that part of the residual was luck, part was coaching, and part was due to other factors not accounted for by the model (officiating for example). With enough seasons for each coach, the luck and 'other factors' would tend to average out, and we'd be left with the additional wins due to 'gameday' coaching.

    In fact, I did this research weeks/months before the Jets game cheating was revealed. Belichick's tenure at NE stood out far beyond any other coach in terms of beating their win expectations (3 standard deviations above the mean). At first, I attributed the results to his coaching genius--going for it on 4th down more often than other coaches for example. Then the cheating revelations hit and made me consider if a lot of those 'extra wins' could be due to an advantage gained by the cheating techniques employed by the Patriots.

    So I didn't have a result and "assumed" cheating. I had a noteworthy result, then cheating was revealed, then I examined how the cheating could explain the prior result.

    On the point about FO DVOA, I'm less inclined to agree. DVOA is an ad-hoc play-by-play scoring system, not a statistical regression model.
    It's clever and novel, but its results are severely over-fit. Their system would be blind to a cheating outlier because DVOA assigns bonus points for success in high-leverage situations. If the Patriots indeed used unfair advantages in crtical situations, their successes would be captured inside the DVOA system. So to DVOA it would always appear that the Pats are winning as many games as they "should."

    Regarding the point about the Colts, I was always taught that adjusting a model to fit an outlier was the opposite of good model construction. Outliers happen. The extraordinary thing about Belichick's Patriots is that they were consistently the biggest outlier every year. The model works really well for 31 agents and the 1 agent that defies it has been caught cheating. I'm not so sure that's an indictment of the regression.

    Here's an analogy to consider. There's a kid in high school who consistently gets B-s on all his quizzes, homework, and assignments. But you notice he gets A+s on all of his final exams. Later, it's revealed he was caught trying to bring crib notes into some exams. Isn't it reasonable to think that the other A+ exams are due to cheating? Is that an "assumption?"

  17. Anonymous says:

    An interesting point I didn't see mentioned is that the second highest 'over-achiever' in terms of wins is Tony Dungy and the Colts. Granted it's not as much as an outlier as the Patriots, but it seems significant.

    1) A cynic would argue perhaps the Colts were cheating too.

    2) Another could argue that dome teams have an inherent advantage in 1/2 their games that over time leads to some over achievement. Perhaps due to crowd noise, real or artificial.

    3) Or perhaps having one of the best QB's in the history of the game tends to lead to more wins than expected because QB's have a higher than normal direct effect on the outcome of game.

    Just a thought. Both Brady and Manning are generally considered to be in that team picture of greatest QB's on their current career paths.

    The two coaches that leap far ahead of any established level of 'excess' wins per year happen to have the two best QB's of the pass heavy era. And one of those coaches drops well down the list if you include his non Brady years.

  18. Brian Burke says:

    nater-Very good point. I'd completely agree, but I don't think NE's pass offense was all that spectacular until this year. In 2006, it was very average but they still won +2 games above what a similar team would be expected to win.

    Also, here is a link to more 'actual/expected wins' charts for other teams. IND is included, and you'll see that they don't really exceed their expected wins at all.

    http://www.bbnflstats.com/2007/06/next-years-wins.html

  19. Anonymous says:

    Completely unconvincing. What does the point about using the information in critical situations have to do with anything? The Pats might outperform their statistics for a variety of reasons including competence, training, and ability to perform in the clutch. Surely the 16-0 exceeded expectations too, post cheating. What a load!

  20. Brian Burke says:

    Ability to perform in the clutch? Are you suggesting the Patriots are so good they only have to try when it really matters? Give me a break.

    Patriots fans--please hide your bias better!

  21. Anonymous says:

    Brian,

    Have you ever considered writing a piece on say, a 38 year old quarterback performing at a level higher than 90% of the league and suddenly playing his best season in 12 years?

    It seems like another statistical outlier and one that can be backed up with circumstantial evidence (38 year old pitchers who shouldn't be throwing 100 mph fastballs suddenly getting exposed as steroid users perhaps?). But why would you write such an article?

    I mean there IS a possibility that Brett Favre is cheating in the unholiest of ways (you know, the kind that requires Congress get involved?), but there is ALSO a possibility that he is just a great quarterback and a great quarterbacks can sometimes accomplish incredible things.

    Brady and Favre will go down as two of the greatest quarterbacks to ever play the game and while there are possibilities of improprieties on their parts and the parts of many other legends, I'd prefer to watch and enjoy their mastery of the position. Not to say that I'd like to turn a blind eye to potential cheating or even go through my football watching life in blissful ignorance; but I'd prefer to believe in the magic of the game and in the players than in the numbers that surround them.

  22. Anonymous says:

    Reading the article and all the comments, I still don't understand the rationale of the Pats' cheating making enough difference to affect wins and losses but not enough to affect Brian's efficiency stats.

  23. Brian Burke says:

    Joe-If they used any advantage only in high-leverage critical situations, the efficiency stats wouldn't be affected nearly as much as the win/loss record.

    Say NE has the ball down by 4 in the 4th quarter. It's 4th and 2. What if the Patriots know the defense's call to overload the left side, and counter with a 3 yd run to the right? NE probably goes on to win, but their offensive running efficiency isn't really affected.

  24. Anonymous says:

    Brian,

    Didn't you use this exact same formula to predict that The Ravens would be a "solid, above-average team in 07"?

  25. Brian Burke says:

    Here is what I said (and since then I've learned that pre-season predictions are a waste of time).

    "They only real difference was the quarterback--Steve McNair replaced Kyle Boller. The biggest changes in team fortunes are due to quarterback changes. We can also note that although Baltimore won 13 games, they had about 10 expected wins. This tells us not to hold our breath for another 13 win season in Baltimore, but still count on a solidly above-average team."

    Unfortunately, the Ravens were forced to revert back to Boller after it became apparent McNair was hiding a pretty serious injury. (Note the sentence above about quarterbacks.) Injuries to their Hall of Fame left tackle, #2 pass rusher, and both all-pro starting cornerbacks may have had something to do with their decline as well.

    I think my main point was that in 2006 the Ravens were really a 10-win team that got a little lucky, and not to count on another 13-win season. Any prediction for 2007 should start with 10 wins rather than 13, and then be adjusted for schedule/personnel changes/etc.

    Sorry that I don't have a regression model for predicting groin pulls and turf toe!

  26. Brian Burke says:

    I need to make a correction to a comment above. I said "Outliers happen. The extraordinary thing about Belichick's Patriots is that they were consistently the biggest outlier every year."

    In some years there were teams with a bigger difference between expected and actual wins than the Patriots. The extraordinary thing is that the Patriots were consistently an outlier on the positive side, without any neutral or down years. But over all years, they were the biggest outlier by far.

  27. Anonymous says:

    "Joe-If they used any advantage only in high-leverage critical situations, the efficiency stats wouldn't be affected nearly as much as the win/loss record.

    Say NE has the ball down by 4 in the 4th quarter. It's 4th and 2. What if the Patriots know the defense's call to overload the left side, and counter with a 3 yd run to the right? NE probably goes on to win, but their offensive running efficiency isn't really affected."

    Brian,

    The part I don't follow is why the the advantages they got from cheating would result in the type of play you describe rather than an 80 yd TD pass for example. Or a mix of both types of plays.

  28. Anonymous says:

    From 2002 to 2006, the Patriots averaged about 2 wins above the expected number of wins. Between 2003 to 2006, their wins-above-expected was consistantly between 2.0 and 2.5. The author of this article hypothesizes that this anomaly was due to their demonstrated habit of illegally videotaping opposing defensive signals. In the first quarter of the first game of this season, the Patriots were forced to cease that habit. All their previous tapes of defensive signals were confiscated. Presumably, they were under close scrutiny by the league all year to prevent them from pulling similar stunts.

    And yet they finished the 2007 season 2.1 wins-above-expected, just like 2003 to 2006. Changing the input variable (taping vs not taping opponents defensive signals) did not change the outcome (number of wins-above-expected).

    So you could conclude:

    a) Videotaping opponents defensive signals did not give them an unfair in-game advantage and is not the factor that explains the anomalous win data (perhaps they were simply storing the info for future scouting purposes, as a record of each opponents defensive play calling tendencies vs down and distance - somewhat more accurate than analyzing the game film)

    b) They were somehow obtaining the same knowledge of opponents signals by other means and using it to regain the same unfair advantage, despite the close scrutiny by the league.

    c) They had and continue to have other methods of cheating that the league does not know about which gives them an unfair advantage. Defensive signal stealing was only a small part.

    Occam's Razor points me to a).

  29. Brian Burke says:

    I'd have to agree, LK. The over-performance again this year would be evidence against cheating as the reason for previous over-performance.

    Then again, this is the biggest outlier season of all outliers. It might not be a good counter-example. It also makes the "luck" theory even more astronomically improbable.

    We're left with these possible explanations:

    1. genius
    2. randomness (luck)
    3. some quality about Brady
    4. cheating
    5. flaw in the model
    6. some combination of the above

    I'd have to say if 5 is true, then the real answer is 1. The other 31 teams all conform to the model relatively well.

  30. Anonymous says:

    Brian,

    One last comment...

    Two facts we know: 1) the Patriots have won games consistantly above their model-predicted wins in each of the last 5 seasons (well beyond expected random variation, apparently), and 2) they were caught illegally recording opponents defensive signals in one game this season. It seems reasonable to speculate that they also videotaped in previous games over the years (although, to be honest, we really don't know if they did, or how long they had been doing it, or if other teams also did it.)

    But if we accept the reasonable premise that the videotaping had been going on for several years and that no other team was doing it, then there does appear to be a correlation between videotaping and winning above expected wins.

    However, as a statistician, you know that correlation does not imply causation. I think another hypothesis that fits the data is this: Bill Belichick is an obsessed, focused, single-minded coaching genius. A symptom of that obsession is the desire to record and have at his disposal every available bit of information, even if recording that information violates the letter of a league rule. The videotaping was not the cause of Belichick's success, but a side effect of his obsessive coaching style.

    Watching the Patriots over the years, I've noticed that Belichick and his team do a lot of the little things extremely well: impeccable clock management; bold, creative, and unexpected play calling on both sides of the ball (such as going for it on 4th down instead of punting); excellent discipline in football fundamentals (such as falling on fumbles instead of trying to pick them up and run); capitalizing quickly on the misfortunes of their opponents (such as a bad call by the refs or a fluke turnover), while recovering quickly when those same misfortunes befall them. All these things might not show up in offensive or defensive stats, but could add up to winning one or two close games a season, where a poorer-coached, less disciplined team would lose.

  31. Anonymous says:

    One last, last comment...

    If you get a chance, compare Ted Williams' famous 1941 batting average (.406) versus the average and standard deviation of the rest of the league that year. Among the 136 players who could qualify for the batting title, Williams' BA was FOUR(!) standard deviations above the mean. This means that, even if there were 15,000 players in the league, Williams would STILL have the highest BA.

    Phenomenal outliers do sometimes occur, even without cheating!

  32. Brian Burke says:

    LK-You're dead right about the correlation/causation. But there is another possibility--it could be a little of both.

    I agree on the going for it on 4th down. I am working on a big project on just that subject for the off-season. If you read the original 'gameday' coach I said some of the same thing.

    And no surprise about Ted Williams--he was a naval aviator after all.

  33. Anonymous says:

    Brian, (no relation)

    I have a few bones to pick.

    1) You state that the reasons the patriots won were:

    a) Luck
    b) good coaching
    c) unknown factors like cheating

    Why not add d) Great players? Underrated players like Mike Vrabel, Tedy Bruschi, and Troy Brown. Then you have pro-bowlers like Ty Law, Seymore, Rodney Harrison, Lawyer Milloy, Willie McGuinnest.

    2) Since when are a team's projected wins *ever* accurate? Every year there is a surprise team in the superbowl. Every year there are a few teams that surprise everybody. When have the analysts *ever* been very accurate with their predictions? What a horrible use of statistics. Basing analysis on people's opinions.

    3) As others have pointed out, turnovers play a huge role in the success of a team. For some reason or another, every surprise team each season has a huge + turnover ratio. Bears 2006, Falcons 1998 are the ones that come to my head.

    4) Using offensive numbers is one thing, but a better comparison is points against for a defense rather than yards given up. IN each of the past 7 years, Patriots have been near the top of the list and points allowed.

    5) In 2001, the Patriots had incredible special teams which turned a number of games (AFC championship game for one!). Please tell me how exactly do you cheat with special teams?

    6) In 2003, the Patriots had an incredible defense. Ted Washington archoring the middle Ty Law, etc, etc. etc.

    7) Then, 2007. With the NFL, media, and fans having the Patriots under the microscope, how exactly did they cheat with 16 wins? A QB that had record in touchdowns with what, only 10-11 interceptions? That's just crazy shit.

    8) If you want to talk about *real* cheating, how about the teams that cheat in free agency and the salary cap. I talked a little about this here in where's the outrage?

  34. Anonymous says:

    flawed argument from the start. Stealing signs is allowed. The rule broken was using a camera.

    Stealing signs by other means even you guys can understand.
    http://sportsillustrated.cnn.com/inside_game/magazine/life_of_reilly/news/2002/01/09/life_of_reilly/

    offensive signals in 2002, but same idea.

  35. Brian Burke says:

    "Flawed argument?" Bill Belichick might disagree with you. Taping signals was important enough to him to risk fines and possibly a suspension. Remember, he was warned previously after being caught.

    Neat article though. I got HD a year ago and was amazed how easy it was to read coaches' lips. During last year's Navy-Note Dame game I could read Charlie Weiss' calls, even the snap count. "It's on three." Too easy.

    And if you watch the NFL Films replay of Super Bowl XXXVI (Rams/Patriots), you can see a sideline discussion between NE defenders of what could only be the Rams' offensive signals.

    Stealing signs, and doing so consistently, would obviously give a team a clear advantage.

  36. Anonymous says:

    I no statistician. I'm not even good with math. But wouldn't the team that makes it to the super bowl have to win a couple more games than "expected" by converting on crucial plays? Isn't that how you differentiate yourself in competitive sports? And wouldn't the fact that the patriots have made it to the super bowl half a dozen times since the late 90's mean they had to win more, throwing off their curve of wins relative to the rest of the pack?

    I could show you a mean graph depicting how a team's winning percentage in football is inversely proportional to the length of their coaches hoodie sleeves - and it would be just as true.

  37. Anonymous says:

    I agree and yet disagree because your stats are quite unreal. The reality is though it is only cheating because he recorded it. Typically NFL teams read signals. They have scouts that their job is to sit and watch the opp and diagram the play. It is a well known fact that everyone does it. The real question is, does taping it make a difference? I heard an interview Boomer about the stealing signals thing and he talked about a game he played vs. the Chargers when they were lead by Dan Fouts. 1 week before they played the chargers cut an offensive player and the Bengals picked them up. The Bengals knew every offensive play the Chargers were running. And according to the interview the chargers won 40 something to 7... Not too important but interesting.

  38. Anonymous says:

    Here's my thing about this. I am a loyal Patriots fan, win or lose. Before Bill was caught cheating, we won some show stoppers, and lost some almost certain victories. I know that you guys are all math and stats etc., but why is it a probability that if Bill was cheating he was sharing what he knew with everyone on the team? Why is it not probable that we just went out there and played our hearts out? Why is it that if "your" team is playing any other team in the league, it's not bitter, but when "your" team is playing the Patriots, everyone despises them?

    I can offer an unscientific explanation. It could be the same reason that all other teams used to hate the Yankees for winning "outside of their means", but now the 86 year losing streak Red Sox are hated just as much.

    Are they all cheaters? You can't fake a home run or a double play...unless you're Barry Bonds. It comes down to this for me: We play hard and break bones. We have injuries that shut down our season...or do they? We play through with as much heart as any other team. We have just as hard a time with your Reggie Bushes and Ladanian Tomlinsons as other teams do. In the "history" of the NFL, the Patriots aren't even that great (he says as a lifetime fan). We sucked for a long time before we got our precious rings. We had an amazing last season only to have it ended by a quarterback with not even half the field vision of Tom Brady. We broke down in the secondary and replaced seasoned center with a rookie that got run over.

    We were pinned with this Dynasty B.S. and have been under everyones microscope ever since Bill was caught. Woul it not be fair to say that most people caught cheating and are punished usually stop after that? If you cheated on a test, or tried to, and were caught, would you keep doing it? I wouldn't. But if you did it and someone else got caught, would you not avoid their mistake to save yourself? (Other teams in the league)

    Did the '72 Dolphins cheat? How about the '85 Bears? One could say they might have, but maybe they were great teams with desire and skill...and luck. I know you keep saying, "It's not speculation because he was caught, we know he cheats", but this year so far we are 5-2. Not bad, not great, but on par with our average season opening...WITHOUT BRADY!

    I respect the research you do and I do accept the fact we messed up. Move on and focus on your team. Who cares. It's a recreational sport guys.

    Fact: Underdogs will win and kings will fall. It's an unspoken law.

    Speculation: The Patriots are a dynasty.

    Food for thought: Let the guys that go out there and play for your favorite team do their job. And when they win, be happy. When they lose, be upset.

    Reality check: None of it matters anyway because until any one of you is manager of a team, a player, or a league official, all of this is trivial.

    And as always...GO PATS!

  39. Anonymous says:

    Listen Brian
    You think your stupid theory proves Belichick and the Patriots cheat. They are better then the rest, that explains everything. Your opinion doesn't matter, and it sucks by the way.
    Churd.

  40. Anonymous says:

    Well, they did cheat, right? That's established. We're just talking about the extent and the degree to which it helped them.

    Have you considered the possibility they could be both good and cheat?

  41. Anonymous says:

    Has anyone ever thought of watching all of the Patriots games for those years, noting crucial plays like 4th and 2, and seeing if there's any actual proof that the opponent run blitzed and the Pats just happened to call a screen pass that play?

  42. Anonymous says:

    a point should probably be made that the patriots coaching is widely known for another method that might be better at explaining your findings. One of the things that is generally recognized as being unusual in the patriot's coaching is their specific concentration on situational practice. according to much news coverage of the patriot's practice is based on repeatedly going over particular moments. i.e. "It's 4th and 3 with 30 seconds left. we need 4 yards for the first down, be in reasonable field goal range." and the play is designed to get precisely those 4 yards. the system that is coached there is largely designed around the concept of "get precisely what we need to get" and is also famous for really being bad for statistics people. the 2007 regular season was an exception to that.

    the patriot's defense in the earlier belichick years was well known for doing 'just enough' to prevent the other team from winning.

    given that perspective, the fact that they have generally outperformed the opposition would appear to have less to do with cheating and more to do with being very well practiced.

    to the last anonymous poster, it might be a point, but it's worth noting that for a lot of brady's first years, the patriots threw screens more than almost any other play.

  43. Anonymous says:

    Since the NFL and everyone else is apparently watching the Pats like a hawk this year, we can suppose that they didn't cheat this year (2008). But did they outperform their stats anyway? Seems like that might be a good test to run.

  44. Anonymous says:

    So, after this year of going 11-5 without Brady for the whole year, Rodney Harrison, Adelius Thomas, Maroney. Losing Asante Samuel to free agency. Bruschi going down for the last 3 games. Adelius Thomas's backup going down for last 5 games. Ty Warren missing 3-4 games, then playing the rest of the season with a torn groin.

    Can you question the brilliance of Bellichick and his coaching staff especially after going 18-1 last year?

  45. Aaron Pressman says:

    Brian,

    I recently came across this post. The Patriots have gone on to win 2.1 games more than your model predicted in both the 2007 and 2008 seasons. Your somewhat tortured hypothesis stated above was that cheating explained the Patriots previous five-year streak of outperforming its predicted win total. It is time for you to admit that your hypothesis has been disproved and the null hypothesis, that the illegal videotaping of defensive signals was not a significant factor in the outperformance, has been validated.

    It always seemed like a shaky tower of tortured of illogic. The cheating advantage was only and consistently used on a few key plays so it was invisible in the offensive and defensive efficiency stats? Somehow no advantage was discernible in 2nd games played against a team in the same season, the exact manner in which the Patriots were found to be putting the information to use -- a missing link that seems sufficient to disprove your hypothesis by itself. And on and on - William of Occam must be spinning in his grave grasping his famous razor.

    And what about your stats? You claim Belichick's average in his initial seven-year run as Pats head coach is almost three standard deviations above the mean. Did you actually calculate the mean by breaking down every stint of every coach at various teams? There's no evidence that you did. And as I'm sure you know, three standard deviations encompasses 99.7% of a normal distribution. Could Belichick be a 1 in 200 or a 1 in 300 coaching talent? Seems quite plausible. Three standard deviations is hardly the kind of impossible level of excellence you make it out to be (Wall Street CEOs were blaming their recent woes on 10 and 20 standard deviation events). All that's assuming, of course, you checked to make sure the coaching averages even fall into a normal distribution. Looks like pretty fat tails to me.

    There is also evidence of bias in your article. Near the top you claim that "No other modern team has even come close to the Patriots in consistently winning more games than their stats indicate" and noting "the 2nd best coach at +1.83 wins per season." Yet, in your prior article looking at the issue, you explain that Belichick's unusual result excludes his tenure in Cleveland. And when you isolated Joe Gibbs' record for his first stint with the Redskins (You call it "'83-'92" although he started as head coach in '81), his average was 2.11. I'll bet big money that if you further looked at slices within his 12 years, comparable to your Belichick run, you'll find a patch of Gibbs that equaled or topped the 2.33 average. Why would you leave out the Gibbs evidence other than to make your argument look stronger than it really was?

    Finally, I realize you are not a journalist and may not understand libel law, but it is dangerous (in a legal sense) not to mention highly misleading to your readers for you to have repeated the Boston Herald's allegations about the 2001 Super Bowl in your post at http://www.advancednflstats.com/2008/02/more-spygate-revelations.html without noting that the paper has rescinded its article and the reporter who wrote the story has admitted that he was wrong. See http://tinyurl.com/d7mo72 for example.

    Please feel free to state whatever moral or philosophical objections you have to Belichick's illegal conduct but please stop trying to argue that you've statistically proven some kind of causal link between that conduct and the Patriots success.

    -Aaron

  46. Brian Burke says:

    Aaron-

    Thanks for the excellent comment.

    1. You're right. I am biased. I have been a big fan of Belichick's approach to the game. You can see that come across clearly in my first post on this subject.
    2. You're also correct that the past two seasons have seen the Patriots continue to defy the model that seems to fit the rest of the league very well. This should move the balance of belief toward 'genius' and away from 'cheater.' I already have a draft of a post to that effect that I plan on posting closer toward the upcoming season.
    3. Unfortunately, he is a proven cheater. Now it's only a question of how effective the cheating was, how far back in went, and whether we can detect it in the stats, which I think we can. I think any discussion of libel is laughable!
    4. Regarding the 'shaky tower of tortured illogic:' I actually performed this analysis several weeks prior to any inkling about the Patriots cheating. At the time I concluded Belichick sure learned a lot between his time in Cleveland and New England, and that he's a visionary and a true outlier. I'm sure you or other Patriot fans would have happily accepted that tower of logic. Then, the revelations came about the cheating. The data now point to a different conclusion in that new light. It's the same tower, same logic, neither tortured nor shaky. Occam compels my conclusion, not yours--We have this outlier result, it could be because of cheating or due to supernatural coaching abilities. We don't need to assume the cheating.
    5. Selectively hunting for sub-samples of over-performance by other coaches is called boot-strapping, and it might be a good idea. But Joe Gibbs was never accused of or caught cheating, so that has to be taken into account.
    6. You also have a good point about the fat tails of the distribution. That might make Belichick a one-in-a-billion guy instead of a one-in-a-trillion guy.
    7. As a business reporter, I'd bet you already know that the financial events are not 10 or 20 standard deviation events. It's more a like a single correlated 3 SD event. The financial wizards just didn't anticipate the degree of correlation.

    Try as you might, Bill Belichick broke the rules, and did it despite being warned. He knew that it must have made a difference and given his team an advantage or he wouldn't have risked it. Even he agrees with me! Like I said above, it's only a question of how big the effect was, and it looks pretty darn big.

  47. Aaron Pressman says:

    Brian,

    Thanks for the lengthy reply.

    You seem to be misreading my comment, however, as some kind of moral defense or denial of Belichick's cheating. The guy cheated -- it's morally reprehensible and wrong. He is not a role model.

    I'm taking issue with your use of statistical analysis and your claim that the cheating in question had much to do with the Patriots' success this decade. You don't have any actual evidence of it and your hypothesis, with all its twists and turns, has now been disproven by the data. You should at the very least make corrections to the two erroneous statements in your post I noted to account for Gibbs' performance.

    Regarding your chain of logic, your initial conclusion was that Belichick's outperformance resulted from superior coaching. No tower of anything, pretty straightforward. It was the same explanation attributable to all the coaches who came out at the top of your initial list and the reverse explained those at the bottom.

    After the cheating was revealed, however, you had to make many more assumptions. Why would the cheating not simply make the Pats more efficient and thus be evident in the efficiency stats? Why would there be no improvement evident in 2nd matches in the same season, the actual way in which in the signal stealing video tapes were used? Why would Belichick try to hide the impact of the cheating from one particular and kind of obscure statistical model (in one of your other comments you note that a more popular model actually incorporated the supposed effect of the cheating into its calculus)? And so on...To get the data and evidence to fit your hypothesis, you had to create a chain of assumptions. It is flatly not the same logic that you used before the cheating was revealed.

    Regarding bootstrapping, my concern is that you cherry-picked a piece of Belichick's record and focused on how it was almost 3 standard deviations above the mean of COMPLETE coaching records. To really know the likelihood of Belichick's outperformance, you'd have to assess all coaches in the same manner, splitting between each assignment and recalculating the mean and standard deviation. The fact that Gibbs, who as you note was never accused of cheating, has a partial-career average close to Belichick's seems revealing.

    I'm puzzled by your argument that a fat-tailed distribution of coaching talent makes Belichick's record one in a billion instead of a trillion. The "almost 3 standard deviations above the mean" makes him at worst a 3 in a 1,000 guy, as I mentioned. In a normal distribution, three standard deviations above the mean exceeds about 99.73% of the set, no? And in a fat-tailed distribution, outliers are far more likely to occur. If pro football coaching talent is not normally distributed, Belichick's result may be even less significant. I believe it was the ground-breaking sports analyst Bill James who said that "absolutely nothing" in professional baseball was normally distributed because players and managers comprised only the very top of the entire talent pool.

    Finally, I did not say that the Wall Street firms were done in by 20-standard deviation events, just that they blamed their demise on those type of events. It's whole other post to explain what went wrong with the financial models of subprime debt (I did get into it some in this article http://www.businessweek.com/magazine/content/08_07/b4071038384407.htm though I'm not totally in love with how this piece ended up). I was using that anecdote to note that a 3-standard deviation event just isn't that impressive and it's certainly not a 1 in a trillion kind of thing.

    I posted my original comment because I noticed that your piece appeared to be widely circulated and cited, including by a New York Times blogger, and I just don't think it holds up. I've tried to keep my comments free of any personal or ad hominem type accusations. My use of the word "bias" was meant in the sense of cognitive bias, perhaps confirmation bias, such as a researcher who becomes taken by a preliminary hypothesis and loses sight of contradictory evidence.

    -Aaron

    p.s. The libel thing isn't a joke. The specific cheating allegation you repeated was retracted and libel law does apply to bloggers. See http://www.eff.org/issues/bloggers/legal/liability/defamation for example.

  48. Brian Burke says:

    If it didn't give the Pats an advantage, why did BB keep doing it after being caught and warned 2 seasons ago? If it didn't give them an advantage, why did BB expend the resources to do it?

    I understood your comment, except I admit I misinterpreted the bias part. But keep in mind this entire analysis was performed in August before there was ever any hint of cheating by the Patriots. In a way, it's the perfect bias-free research. I'm only re-examining the results within a new context.

    The analysis here has no twists, no turns. It's a straightforward multivariate linear regression of independent statistics known to estimate team season win totals very accurately. It's something any undergrad stats guy should be able to grasp--that is, it's very straightforward with nothing to hide, no "hand waving" as the professors say. There was an extreme outlier, not only in the fact that the aggregate multi-year estimation error was through the roof, but that there was never a single negative or break-even season.

    So we essentially have this multiple choice question. There's this unbelievable outlier--what explains it?
    A. Luck? Well, maybe, but the odds are astronomically against it.
    B. Coaching genius beyond anything a mortal human could ever comprehend? Believe that if you want, Pats fans!
    or C. Cheating? Something we already know as fact.

    Belichick cheated, most likely for years and in more ways than is public knowledge. The question I tried to answer is: can we detect it in the stats. And the answer is a clear yes.

  49. Anonymous says:

    I like the data, but your interpretation of this data is mostly speculation and little fact. Frankly, parts of your argument sound just as biased and irrational as the Pats fans who are blindly defending Belichick. If your answer is a clear "yes," then why does Belichick continue to buck the trend? the last two years put your data in a new context. Either way, your old argument was pretty shaky.

    Your argument is built on the premise that Belichick would only steal signals in "critical" situations? Basically, cheating just enough to gain a significant advantage, yet not enough to disrupt the stats. That's an awfully fine balance, is it not? How would Belichick know this happy medium, and consistently find it year in and year out? And why would Belichick limit his cheating to only a few critical plays - thus limiting the effect of his work? It's like having all the answers to a 100 question test and only using the cheat sheet for just 3 or 4 questions. It doesn't add up. Wouldn't he be tempted to cheat more in losing efforts, thus skewing the data? Also, why allow the game to progress into "critical" situations at all, if you have tools to prevent it?
    Also, how many times a year do games actually boil down to such "critical" situations? Not only that, but how many times does it happen against an opponent Belichick already taped?

    Also, you've done a poor job of defining what is a "critical" play. In theory it sounds nice. In reality? Well, you are assuming that Belichick can identify the few plays on which a game will hinge - within the heat of the moment, no less. That ability alone is highly impractical. Kind of like your description of "genius beyond anything a mortal human could ever comprehend."

    I'll ignore other barriers to cheating like, dummy signals and the sheer logistics of deciphering, countering, and relaying said info to the QB within an extremely limited timeframe.

    Finally, I have a bit of a problem with your refusal to accept that other coaches have cheated - while at the same time assuming the extent of Belichick's cheating runs far deeper than we know. You can't have it both ways. You can't take a 'just the facts' approach when it suits your argument, and speculate when it doesn't. Saying that it is not "relevant" if other coaches cheated is really just as ignorant as the Pats fans who claim "everybody" did it!

    Any fan looking at this from a dispassionate perspective can see that spying was/is pretty common in the NFL. Even if we as fans think it is morally reprehensible, it's pretty clear that the coaches - even the ones who did not cheat - weren't very concerned about it. Both Jimmy Johnson and Barry Switzer publicly admitted to taping signals. Jimmy said he was told how to do it by a Chiefs scout when he first entered the NFL. He also said that the best known signal stealer is Howard Mudd; offensive line coach for the Colts. Howie Long said he knew of this practice too. And we have proof that Herm Edwards waved to the camera and said "Hi Bill!" I mean, Belichick didn't even bother to hide his cameraman. He was on the opposing sidelines, dressed in full Patriots gear, filing away. His actions were as overt as can be. This nonchalant attitude - by all parties - points to the fact that these coaches simply saw it as being part of the game. Even the innocent coaches knew it was happening and turned a blind eye to it for decades. They merely took steps to hide their signals.

    So, excuse me if I don't get all up in arms about this. I suspect most fans have come to the same conclusions I have - unless they want to stick it to Pats fans, of course ;) Your data was probably the most damaging evidence against Belichick -- right up until the past two years where we see the trend has continued. That blows a pretty big hole in your theory.

  50. Anonymous says:

    A couple of observations: you're assuming videotaping= competitive advantage..You reason: why would BB risk getting caught and expend resources for something that did not lead to some sort of advantage.. Very logical conclusion.. However, as someone already mentioned, it's quite possible that videotaping was just a manifestation of BB's obsessive nature and the advantage gained if any was minimal. If videotaping was such an advantage why as someone has already mentioned didn't the Pats use the advantage in ALL situations and not just in crucial moments of a game as you hypothesize to explain away the efficiency stats problem? Also, Jimmy Johnson admitted to doing the same thing and along with Dick Vermeil have publicly stated that videotaping offered no advantage..My question to you is have you looked at JJ's coaching history and his teams expected wins vs. their efficiency? do they also have the same inflation as BB's Pats? If you expect videotaping would lead to some competitive advantage, you would also expect to see similar win total inflation from JJ's Cowboys/Dolphins.. The other question would be have you look at BB's Cleveland numbers? I would assume he was also videotaping back then, and as stated, you should see the same win total inflation IF videotaping= competitive advantage..

  51. Brian Burke says:

    Fair points, but you're assuming a lot there. Belichick's obsessive nature, that he videotaped in Cleveland, and that the taping was the only form of the cheating. I don't think that answers the question, why did he keep doing it after being warned, knowing it might bring a high cost to himself and to his team?

  52. Anonymous says:

    Bill Belichick = Wizard of Oz.

  53. Anonymous says:

    But even if Belichick still is cheating (in another form)...doesn't that still rule out the effect of taping signals? If taping signals explains - or even partially explains - his sucess, then his numbers would have taken a hit regardless of other cheating.

    Yet, this didn't happen. So can't we conclude that the act of taping signals was of little significance?

  54. Anonymous says:

    how can you assume that it must be luck, genius or cheating? does your analysis take into account EVERY other variable? it seems to me that the only thing you have proven is that there is a certain aspect of belichick, whether it's his genius, or a certain strategy used at a critical point in the game, or training technique that maximizes the abilities of a player, etc., that is unique to him and only him. and it is this "thing" that gives his team that competitive advantage. can it be cheating? sure. cheating that gained an advantage could certainly be unique to belichick. but how can you discount every other possibility?

  55. NEKID says:

    It is January 3, 2011. NE's record is 14-2, in a year they started rookies, cast-offs, traded Randy Moss, signed Deion Branch, and lost significant players on both the offensive and defensive sides of the ball. In this year, when most analysts accused Belichick of 'giving up on his season' in trading Moss, NE has fielded a team which has been a high-scoring turnover machine, with a 1000yd rusher, and a QB who's TD/INT ratio is 9-0. Explain THAT for me, will ya...?

  56. Anonymous says:

    None of the Patriot defenders can explain why the stats don't even out over time. The odds are highly against this. I know it must suck to think your team didn't really win those super bowls, but they were caught cheating, and this explains the statistical anomaly better than luck. As for the stats continuing after the scandal, maybe they're still cheating.

  57. Anonymous says:

    Before I can comment on the results, I really need to understand the methodology and assumptions made - because that's at the crux of the argument. But none of this was in the article.

    I'll deal with one area a little more in-depth. In a link you mentioned "passing, running, and turnover efficiency, plus penalties". Of course, the Patriots have historically statistically given up much fewer points per yard than the average team by using what most call a "bend but not break" defensive philosophy. If you're using yardage as a significant measuring stick, then do you do anything to account for this defensive philosophy other than rationalize an answer of probable cheating? If so, I haven't been able to find it.

    I would hope that the differences between expected results and actual results would first lead you to examine the assumptions and methodology you used...

  58. Anonymous says:

    I am curious about something that was said in the original article and then repeated in the comments. That is, Belichick was "caught, warned, but continued."
    Maybe I'm missed something, but if the Sept 2006 memo to all teams was the warning, when were the Patriots caught before this?
    I am aware of a Patriots cameraman being removed from a game against the Packers at Lambeau in 2006. There was also a Jets cameraman removed from Gillette that year. As far as I can tell, nothing happened, discipline-wise, for either of these incidents. If the league warned the Patriots beyond the 2006 memo, I never heard this brought up after the story took off in September 2007.
    If there was another warning by the league in addition to the '06 memo, I'd be interested to know what it was. I thought I had read a lot on this, but I'm willing to learn more.

Leave a Reply

Note: Only a member of this blog may post a comment.