MIT/Sloan Panel

Here is the video of the panel at the MIT sports analytics conference I referred to in my post about how Bill Polian doesn't get it. The discussion is educational throughout, and despite my criticism of Bill Polian, he has some very wise things to share. The participants are Mark Cuban, Jonathan Kraft, Daryl Morey, Polian, and Bill Simmons, with Michael Lewis as the moderator. It's over an hour long and worth your time. I've embedded it below, but first let me share what I took to be the most interesting points:

1. Sports analytics are here to stay. Get used to it. As an owner, GM or coach, get informed or pay the price.

2. In baseball, it's all about measuring individual ability. In basketball, it's about measuring how various combinations of lineups work together. In football (until this year), it was about financially squeezing under the cap. In all sports, it's about finding the "undervalued asset."

3. Teams are aware their formulas and methods aren't going to stay secret very long. Analytical staff come and go and take expertise with them.

4. Bill Polian thinks the Holy Grail of football analytics is figuring out how a player who thrives in one system (a 3-4 for example) would fare in another system, say Tampa-2. And how much should he be paid when going from one system to another?

5. Polian says a former player or "military officer" with analytical expertise is needed to be credible within the football culture.

6. Analysts need to write in simple terms. The onus is on them to explain what they're doing. "Speak English, please."

7. Mark Cuban wants standardized league-wide enhanced play-by-play data. A lot of the expense in analytics is in generating the level of detail in the data needed for sound analysis.

8. Football player evaluation has to be far more subjective due to the nature of the sport. (I'll have more to say about this soon, but in short I believe the best outcome is in a fusion of subjective evaluation by experts --filtered, standardized, validated and exploited with statistical analysis.)

9. Regarding the 4th down analysis Polian says (getting fairly worked-up), "There is ZERO out there that's any good." It's completely "worthless" because football is so "technical" and "team oriented." Plus, different teams have different systems, according to Polian. This is a non-sequitor wrapped in an error shrouded in a fallacy. Mr. Polian is a smart guy, and I bet if I had 10 minutes in front of a white board with him, he'd buy in.

10. Everyone seems aware of the dangers of making conclusions from small samples.

11. Many teams that are using advanced stats hide it or deny it for PR reasons.

12. Team executives who are reluctant to embrace analytics aren't necessarily dumb. They're just wise. It's not always good to embrace things you don't fully understand. Not everyone should be expected to have the technical background needed to get this stuff. Just ask the investors in complex derivatives from Wall Street.

13. Right now is a special time. Teams can create an advantage with good analytics, but one day, once everyone has caught up and all teams have the best methodologies, they will all be back on the same level playing field. (Now is the time to take advantage.)

14. Both Kraft and Polian were very focused on how important it was to find players who "fit their systems." (I would bet this notion is far overblown. Show me some data.)

15. Bill Parcells is not terribly bright. (See Kraft's comment near the 36:45 mark.)

16. Exploitation of referee/umpire tendencies may be the next big thing, if it's not already. I'm sure that's all done behind closed doors. Cuban's comments make it clear this is particularly important for the NBA in which outcomes are so susceptible to one or two calls. (The NBA has a huge sample size to work with too.)

17. Psychological testing is big for the Patriots and Colts, and probably many other teams.

18. Polian and most everyone except Rockets GM Daryl Morey (the one guy with an analytical background) believes in "clutch."

19. Polian said something interesting about player selection. "The best of us are batting .550."

20. No one believes in injury prediction.

It seems to me that sports executives ignore the advances of statistical analysis at their own peril. But I believe it's a two-way street. Analysts should listen and learn. A lot of what the old-school guys say is still bunk, but there are nuggets of wisdom to be found.

My thanks to reader James for providing the link.

  • Spread The Love
  • Digg This Post
  • Tweet This Post
  • Stumble This Post
  • Submit This Post To Delicious
  • Submit This Post To Reddit
  • Submit This Post To Mixx

11 Responses to “MIT/Sloan Panel”

  1. Anonymous says:

    Did Bill Simmons have anything to offer at all?

  2. Anonymous says:

    Well, for one, he thinks he'd make a damn good GM of the Timberwolves

  3. Buzz says:

    Not to hijack this thread as it is a very good one but it got me thinking about a topic….How much credit should each position get for the EP on a play? I came up with an idea that might have some merit but clearly needs some work so I thought I would throw it out there and see what some of the smart posters here thought. Most of the assumptions in this post will assume that NFL teams are rational and pay for their assets appropriately. Obviously this isn’t always the case but hopefully in total it will be close. I think this will have to post in a couple of posts.

    The biggest assumption I am going to make is that we can use the amount of salary cap dollars spent on a position to determine its worth. Using this assumption I set out to see how much was spent on each position. As it turns out the NFL allocated approximately $1.7B of its total salary cap to offense during 2009 with the following %’s spent on each position.
    RB 14%
    WR 21%
    TE 8%
    QB 22%
    OL 35% (in the future you could break out RG, LG, LT, RT, and C)

    Obviously some of this money is spent on 2nd and 3rd string players and if we want to allocate the EP of a specific play we need to figure out how much money is spent on each player that is on the field so we can divvy up the amount appropriately. So what I did was determine how many players were on the field at each position during an average play. To do this I went to and used their snap count to see how many snaps were assigned to a QB since outside of the random wildcat plays a QB is on the field for every snap. I then compared the number of snaps at RB, WR, and TE to see how many were on the field for each QB snap. It turns out that there are approximately 2.5 WR’s, 1.4 TE’s, and 1.1RB’s on the field each play.

    If you assume owners are paying their players rationally then the salary allocated to the top 81 WR’s (2.5 players on the field * 32 teams) would be the amount of money spent on wide receivers if they never had to come off the field for injury, rest, etc. If you repeat this for RB’s and TE’s you get the top 35 and 44 players respectively. For OL I used 160 and QB you would obviously get 32. The total of this money spent on “starters” is approximately $1.33B and is divvied up by position as follows. Note that the amounts only change slightly from the first calculation with OL getting the biggest bump, which I think seems fair.
    RB 10% $133M
    WR 22% $288M
    TE 7% $98M
    QB 23% $303M
    OL 38% $511M

    Now that we have a general idea of how important each position is to the league average team we can start making some assumptions based on these values. Since the use of a play (down) is a measurable commodity we can assume that a team will value all plays equally. If this is a valid assumption and an average team passes on 56% of their plays (as they did in 2009) we can assume that a team would spend approximately 56% of their $1.33B of starter money to optimize their passing game and 44% of their money to optimize their running game. This equates to approximately $751M of starting salary cap space spent on the passing game and $584M on the rushing game.

  4. Buzz says:

    So which positions do teams pay to make up $584M of money spent on the running game? RB’s are only paid $133M and part of their job is pass blocking and catching passes. The O-Line is paid $511M but they don’t just block for running plays they also block during passes. One could argue that all $303M of a QB’s cost should be tied to the passing game but I don’t agree with that because a good QB is creating value in the running game by keeping safeties out of the box, etc. In this same light a WR occupies corner backs/safeties and during runs and also blocks. Think of it this way, if a WR wasn’t on the field there would be another defender to tackle the RB. I don’t have a full proof answer of how teams spend their money but here is my first subjective thought.

    Position Running Passing
    RB 70% 30%
    WR 15% 85%
    TE 40% 60%
    QB 15% 85%
    OL 70% 30%

    With these weights I come up with following amounts spent on each player by play type
    Position Running $(M) Passing $(M)
    RB $93 $40
    WR $43 $244
    TE $39 $59
    QB $45 $258
    OL $358 $154
    Total $579 $755

    Or shown as a percentage:
    Position Running % Passing %
    RB 16% 5%
    WR 7% 32%
    TE 7% 8%
    QB 8% 34%
    OL 62% 20%
    Total 100% 100%

    So basically what I am arguing is that for each Expected point added on a running play 16% of that value should go to a RB, 7% to the WRs, 62% to the OL, etc. A couple of caveats with this - First not all teams value every position equally. For now we will ignore this but you could make team adjustments later.

    Secondly, I think this is probably a pretty good way of allocating EPA at the end of the season if you didn’t have play by play data to help you with but since we do there has to be a little better way. The reason why this isn’t necessarily great on a given play should be fairly obvious if you consider a RB reception. In that case the running back would be given significantly less value than the WR’s on the play. Over the course of the season this would hopefully even itself out but wouldn’t be 100% accurate. What we need to do is get a value for on the ball players and a value for off the ball players. We will come back to this in a second.

    Secondly, EP is basically a league average. If we really want to determine how much value a player created we should compare the EP gained on a play to the EP that would have been gained on a play by a practice squad (replacement level) player.

    Footballoutsiders have concluded in the past that a team of replacement level players would be outscored 260-407 for the season. However, of this value some of it is attributable to special teams thus they concluded that a replacement level team would score 274 points on offense and allow 394 points on defense. Since a league average team (one that you would expect to have an EP of zero for the season on offense) scored 352.5 last season so we can assume that a replacement level team would lose approximately 78.5 vs a league average team. Note, in reality I am guessing this is actually higher since we know that coaches don’t always operate in the most effective way and lose EP points on 4th down decisions, etc. Maybe Brian would know how many league wide EP points are really available.

  5. Buzz says:

    To convert these EP plays to a per play basis we need to know how many plays would it take for a replacement level team to lose these points. In 2009 the league average team ran 1,007 plays. But I think it is safe to say a replacement level team would have more three and outs thus significantly less plays. How many plays does the bottom 10% of teams average over the last few years? Approximately 940. In addition the 2002 texans had 947 plays and the 1999 Browns had 865 plays. Since they are some of the newer teams in the league I will guess a little lower than 10% and say 930 plays. If we assume that a replacement level team can pass and run equally well we can divide 78.5 points by 930 plays we can see that a replacement level team will average -0.084 EP per play.

    Disregarding caveat #1 for a second, now we can estimate how much value each player added on a rush attempt with an EP value of .4. We start by adding the replacement level value (.084) to the EP to give us an EP above replacement player of 0.484. Now we can multiply this amount by each of the percentages above and determine the following values created by position and by player (assuming 2.5WR’s etc).
    Position Total Value Per player %Per Player
    RB 0.078 0.078 16%
    WR 0.036 0.014 3%
    TE 0.033 0.024 5%
    QB 0.038 0.038 8%
    OL 0.299 0.060 12%

    To do this calculation effectively we would have to mirror up all player participation stats with EP for all plays. Or alternatively, we can shortcut this by following these steps.
    1. Determining the value of all plays where the player in question actually touched the ball and was included in the play by play database.
    2. Determine the total value of all plays that the player was not involved in.
    3. Determine what percentage of those points go to that players position.
    4. Determine what percentage of non-ball carrying plays from that position the player was involved in
    For example, let’s say player A was a targeted wide receiver on 100 pass attempts and participated in 900 plays. The team ran 1,000 total plays which included 2,500 “WR plays”. During his targeted plays it was determined that he should get credit for 5EP. In addition it was determined that WR’s should get additional off the ball credit of 10EP. He could then be given 3.2 of those off the ball points, ie (900-100)/2,500=32%*10EP=3.2EP, and if you add them to his on the ball points of 5 and determine that his true value was 8.2 EP.
    So what do you think? Is this a reasonable approach to start looking at a players value or is it not realistic at all? What ideas would you have to make the idea better? I have some thoughts on how much a target is worth (23% of a pass value for a WR down to 15% for a RB), how much an EP is worth to help determine a players salary that he earned, how much more a team pays a WR1 to be a “threat” vs a WR2, etc but it all sort of hinges on some of these calculations so any feedback would be greatly appreciated.

  6. Jonathan says:

    I bet points 9 and 11 are related.

    "19. Polian said something interesting about player selection. 'The best of us are batting .550.'"

    I believe that. The best gamblers bat .550 against the spread. I'm not a gambler (for that reason more or less), but I am aware that the best of the best, in effect, are looking for matchups where the line is off by 3 points at the most. That's why Vegas odds are typically close to actual probability.

    "Everyone believes in clutch except for that one guy."
    I probably believe in clutch, but I think the "clutch gene" is a much smaller component of "clutch performance" than most people make it out to be. It might turn a .310 hitter into a .320 hitter. The problem I have with clutch is that it's so horribly misdiagnosed and overrated. If you have rings, you're clutch, no matter how good your teammates were. If you don't have rings, you're a choker, no matter how many rings you lost due to factors outside of your control. Peyton Manning is considered a choke-artist again because of:

    1) A miscommunication on one throw against the Saints. Talk about jumping to conclusions because of a small sample size.
    2) That onside kick.
    3) Drew Breeeeeeeeeeeeesss

    Anyways, I think most owners are wise enough to let individual performance determine whether a player is clutch, not team accomplishments/small sample sizes. So I'm not ripping on any of them.

  7. Doctorjorts says:

    A very interesting post that shows the "stat guys" what things look like from the other side of the owner's box glass. I'm a medical student, and I can see a lot of similarities here with a patient interaction - it doesn't matter how demonstrably right you are in your analysis, if you want an owner/patient to believe in what you have to say you have to say it on their level. Numbers and statistics or cutting edge medical science might be where you "live," but all the statistical power in the world doesn't mean jack squat if no one understands or trusts you.

    I disagree that football "systems" hold little value with regard to player development/performance. I think from a statistical point of view we tend to try to simplify players to single points of data when there are many multiple skills required for each position. At the very least, there should be three different factors that play into making a good athlete in any sport: strength, speed and "sport IQ"/instincts (the knowledge, conscious or not, of what to do in a situation regardless of your ability to execute). The degree to which a system emphasizes the importance of those skills and their subcategories would have a wildly variable impact on player performance, IMO. The ability to predict that impact may be impossible, but the impact, I think, would be there nontheless.

  8. Danny T says:


    11. Many teams that are using advanced stats hide it or deny it for PR reasons.

    I'm obviously biased, but just look at the Niners. Paraag Marathe is basically a pariah among the fan base...for no apparent reason except for the fact that he's the resident "stats guy." Writing on Niners Nation, there's a significant portion of the readership that simply thinks statistical analysis is akin to voodoo.

    To me, the situation with sports stats is eerily similar to what I've encountered in my other life as a sport psychology grad student. There's a ton of really bright people out there, and a ton of research with concrete findings, yet the vast majority of upper-echelon coaches in big-time college sports won't let a sport psychologist near their players. Sport psychology basically isn't the way the ol' ball coaches did things, and so goes sports stats. Which leads me to agree whole-heartedly with this...

    1. Sports analytics are here to stay. Get used to it. As an owner, GM or coach, get informed or pay the price.

    The more teams like the Patriots, Colts, and Eagles succeed consistently, and the more young stat geeks decipher the statistical codes of sport performance, the more they simply can't be ignored. The on-field results, and statistical/psychological work that influenced those results, will serve to make the validity of these things beyond debate. 30 years from now, the idea that sport stats and sport psychology have a positive influence on performance will be the ultimate "duh" least in my opinion. It's up to those of us who believe this sort of thing to move onward and upward despite the bad PR we might currently engender.

  9. Anonymous says:

    The fourth down discussion is always interesting to me. I think all it takes is a coach with nothing to lose to go for it for a season. I'm really surprised that this hasn't happened, think Jim Zorn in 2009. He had to get deep into the playoffs to keep his job. In his situation there was much less incentive to be averse to risk.

    Once we see one coach in the NFL or a major college program using a much more aggressive approach, there will be at least some sample to look at. There's a number of coaches who are going to be coaching their last year. If you want to persuade someone, that is the guy to talk to.

  10. Anonymous says:

    First of fall, sorry for my mistake i'm not used to write in English

    A comment on fourth down : We all agree that on average it's better to go for it because you have 30-40-50% or whatever chances to convert it and that makes up for the times you don't make it ;
    But also know random events can come in quite some large chunk i.e. if you flip a coin 100 times it's not uncommon to have series of 5-6 heads or tails.
    futhermore in football each situation is very different : 4th down depending of distance to go and on the field, vs 3-4 or 4-3 defense, ect, ect.
    So each 4th situation situation comes in very small sample over the course of a season : i'm sure we can regroup,gather some because they are "sufficiently alike" but is it then 10 different groups of 5 similar 4th down in one season? 2 groups of 25? or 50 groups of one ?

    So my point is : it is possible - albeit unprobabl - that given the small sample size a team fails on every one of is 4tha down attemps
    over the course of a season, altough it is less possible if the "same" situation happens 50 times in a season than 2 times.
    So punting the ball becomes an assurance vs "bad luck", whereas you can still go for it when the benefits are big enough or when the particular situation on the field is in your advantage.

    Dose that makes any sense? Is th sample size really to small?
    And if it does is there a way to quantify that? Meaning how manny particular 4th down situation an average team faces during one season?

  11. Anonymous says:

    Bill Polian proves that being a GM is so complicated that you can have blind spots where you make very sub-optimal decisions, and still be superior enough in other areas to overcome those specific inadequacies.

    In football, finding players that fit your system is probably very important, but not nearly as important as developing a system that allows you to maximize performance relative to cost. The Broncos passed on Dez Bryant for Demaryius Thomas because of 'system'. If you must pass on far superior players because of your system then you are not maximizing your potential output. The genius of Polian is knowing that passing offense is the most important aspect of winning and the most consistent year-to-year and he therefore puts his resources there.

    Polian's answer on the 4th down thing is so ridiculous it borders on the absurd.

    When you use the phrase "Polian doesn't get it" you really need to be referring to the decision to tank Weeks 16 and 17.

Leave a Reply

Note: Only a member of this blog may post a comment.