## Predictability

Over the past few months I've been writing about how game theory can help us understand play-calling in football. Not only can it help us understand why coaches call the plays they do, but it can instruct us on what the optimum balance of play types should truly be. Offenses always need a mix of strategies to maximize their gains, no matter how much better they might be at running over passing or vice versa. But just important as the ratio of the strategies is the unpredictability of each call.

The mix of plays needs to be random to be effective. That's not to say play calling should be picked willy-nilly out of a hat. For every situation there will be an optimum ratio of play types. For example, on 3rd and 1 situations, teams should generally run at least about 85% of the time. But within that 85/15 run-pass mix, the decision needs to be unpredictable, which means it must be random and independent of the previous play call. The problem for play callers, and the opportunity for defensive coordinators, is that people are terrible at randomizing.

There's a story about a statistics professor who challenges his class to a contest. He divides the class in half and tells one group to flip a coin 100 times and write the sequence on the board-- THHTTH... The other group is told to invent and write their own sequence of heads and tails on the board as randomly as they can, without looking at the other group's sequence. The professor says that if he can't tell the true random sequence from the fake one, he'll give everyone an A (or something). He leaves the room until both groups are done, then returns and instantly spots the fake sequence.

The professor can identify the fake random sequence so easily because it has too many alternations between heads and tails, and too few long streaks. The fake sequence looks like HTTHTHHTHT, while true randomness often looks like HHHHHTHTTH. True randomness can be quite streaky (which is partly why people fall for fallacies like "being in the zone" or "the hot hand").

If I'm a defensive coordinator, I'd like to know what kind of play the offense is going to run. I don't need absolute certainty--any idea is better than no idea. For example, for all 2nd and 10 plays in the NFL, offenses run the ball 46% of the time. But what if defensive coordinators could know that based on other circumstances, this particular 2nd and 1o will be a pass 80% of the time?

Take a look at run-pass balance on 2nd down situations. The graph below shows the percentage of run plays on 2nd downs according to the yards-to-go situation. There is one noticeable aberration at 2nd and 10: runs are far more frequent.

Why would runs be far more frequent on 2nd and 10 yards to go than on 9 or 11 yards to go? The task facing the offense is not meaningfully different. What makes 10 yards to go so special?

The key is that 2nd and 10 situations are several times more likely to occur due to an incomplete pass than due to a run for no gain. This suggests that, effectively, teams are significantly more likely to run following a pass than pass following a pass. Therefore, offenses are not randomizing as much as they are alternating.

Play calls are not independent of the previous plays, even for similar down and distance situations. NFL offenses are therefore substantially, although not completely, predictable.

Instead of HTHHTHTH in a statistics class, we have RPRRPRPR in football. The patterns remind me of a language with consonants and vowels. But play calls are not simple either/or run or pass decisions. There are several variations to each, just like there are a's and e's and b's and c's.

Linear B was an ancient written language found in Crete and named for its straight lines. It was a precursor to the Hellenic Greek language dating back to the times of the Homeric epics. Several tablets with etchings in Linear B were excavated in Knossos, thought to be the capital of King Minos. The script was completely unlike any other, and baffled archeologists for decades. Researchers had nothing to go on except the patterns found in the writing.

In the mid-1940s Alice Kolber, an American professor, theorized that the characters represented syllables and that the language was highly inflective (having lots of different conjugations). Then in the 1950s, an amateur archeologist named Michael Ventris cracked the code. He found that each character represented a consonant-vowel combination. Each sound a person can make either goes well with others or it doesn't. This was all that was needed to eventually decode Linear B and unlock all its secrets.

Just like vowels and consonants, runs and passes tend to alternate. And certain types of plays tend to work well before and after others, just like "th" or "rn" or other consonant combinations.

I'm not claiming that we can crack the code on play-calling any more than you can predict my next word. My point is that a serious cryptographic analysis of play-calling could reveal tendencies not previously thought possible. For example, try to predict the next letter of the word "th..." Chances are very good it's a vowel, and if it's not, it's got to be an 'r.'

I'm sure coaches pore over hours of film trying to discern opponent tendencies, and are looking for things I couldn't even fathom. But it seems that they are focusing on situations in isolation. They're zeroing in on observations like, "they run on 2nd and long 35% of the time in the red zone." Apparently, coaches are not picking up on the fact that the same team might run 65% of the time in the same situation following a pass. If they were detecting these patterns, they wouldn't let their own offenses be so predictable.

I realize this is a wondering essay. My main points are that:

1. Offenses need to be unpredictable to be effective.
2. Plays need to be random both with respect to previous instances of the same down/distance situation and with respect to previous plays.
3. NFL offenses show evidence of patterns, even when holding for situational effects.
4. Coaches don't seem to be aware of the patterns, and they can be exploited.
5. And lastly, the only true countermeasure is to somehow inject genuine randomness into play calls.
I was going to finish this article by retelling an Edgar Allen Poe story in The Purloined Letter. But as I researched the details of the story, I realized I was beaten to the punch by the Smart Football blog. Check out this article on Poe, rock-paper-scissors, and play-calling.

### 30 Responses to “Predictability”

1. Anonymous says:

Could it be that teams run after a pass attempt (and vice-versa) because certain of their players are tired after the first attempt, and they want to focus on less-tired players?

No idea, just a thought.

2. Anonymous says:

Yes, you are correct that randomness can be streaky, but that is no reason to call "the zone" or "the hot hand" fallacies. I think it's silly to just ignore or excuse the unmeasureable factors of the game like that.

3. Anonymous says:

Yes, you are correct that randomness can be streaky, but that is no reason to call "the zone" or "the hot hand" fallacies. I think it's silly to just ignore or excuse the unmeasureable factors of the game like that.

4. Anonymous says:

It's been pretty well accepted that belief in the "hot hand" arises from a misconception of randomness as described here. There's also been a study on the hot hand by psychology departments at Stanford and Cornell: http://wexler.free.fr/library/files/gilovich%20(1985)%20the%20hot%20hand%20in%20basketball.%20on%20the%20misperception%20of%20random%20sequences.pdf

5. Anonymous says:

6. Anonymous says:

Thanks for posting that Chris. I do see where they're coming from, but if you have ever played sports you will most likely know what it's like to "get hot".

In the case of a QB who is "in the zone"...the more successful plays are run, the higher the confidence of the QB, which when controlled can lead to a higher percentage of completed passes. Scientists can run all the studies they want but no numbers will ever prove to me that players can't get on hot streaks.

7. Anonymous says:

Sorry for all the posting -

It all goes back to the problem I see with a lot of stat sites...if you can't measure it, it either

A. doesn't exist

or

B. isn't important

And that's fine, everyone can have their opinion, but it just seems like a cop out. To say that players can't grow in confidence as they make more successful plays is crazy. Anyone who has actually played sports would know this, and yeah, you can't really measure a player's psychological strength but the signs are there and shouldn't be ignored.

Sports, to me, are just as much a mental game as they are physical. You can measure the physical much easier, but you can't ignore the mental.

8. Anonymous says:

Scientists can run all the studies they want but no numbers will ever prove to me that players can't get on hot streaks.

This is obviously wrong - personal experience-based arguments are laden with pitfalls and biases; specifically, that people remember when they hit three shots in a row and that the fourth one went in, not the other time that they missed the fourth one. Specifically, claiming no contrary evidence will ever convince you of something is just not reasonable. I've played sports my whole life and I've felt hot, just like the NBA players in the study felt hot - but I can also recognize objective contrary evidence when it's there, and I realize that my own personal experiences are far less reliable than properly used data.

Back to the article: Yeah, this seems really important. It seems intuitive now that I see the data that this is what has been going on, and the findings seem clearly applicable to both offenses and defenses (who, if they realize this, can step up the run defense following incompletions.)

9. Anonymous says:

Don't get me wrong - I'm not saying that every time you hit three shots in a row you're on some hot streak. What I'm saying is that hot streaks exist, period. Analysis that ignores the mental aspects of the game is just as laden with pitfalls and biases as some experience-based arguments. There are principles, rock solid psychological principles, that are in effect that absolutely MUST be taken into account in order to create the strongest analysis.

And, I might add, saying that no amount of contrary evidence will ever convince me of something is absolutely reasonable. Objective contrary evidence is only useful when tempered with personal experiences, and vice-versa.

But, kudos to Brian and everyone else out there who do the best they can without that information. None of us can sit down with T.O. and ask him how he's feeling, so we're kind of handicapped in that way.

10. Anonymous says:

Being fooled by your own success into thinking you are on a 'hot streak' is no different than mistaking the runs in the coin-toss experiment as non-random (which is what most people think). And I'm not judging, this happens to EVERYONE: Emotional responses are often irrational.
A study was done some years ago examining the free throw tendencies of Larry Bird (pretty good player) and a few others from that era. If I recall correctly the finding was that there was no statistically significant link between subsequent free throw attempts, i.e. they were random results. Remember that with a highly skilled player who succeeds at something 8 out of 10 times he is expected (following the coin toss logic) to have some pretty lengthy runs in whatever sample of attempts you take. In my opinion the 'hot hand' is a mixture of skill, randomness, and misconception.
See "Fooled by Randomness" for more on mistaking chance for something with greater contextual meaning.

11. Anonymous says:

I agree with you, to a point. If I make ten free throws in a row, I KNOW for a fact that it's luck. But if Larry Bird makes ten in a row, the luck:skill ratio is a bit different, don't you think? And obviously just because he makes ten in a row, that doesn't mean that #11 is going in, but I think you have to consider that after #10 he's feeling confident and he is more focused and there may actually be a greater chance of #11 going in.

So yeah, the "hot hand" is a mixture of skill, randomness, and misconception.......but I don't think you give those who have more skill than others the credit they deserve for getting on hot streaks.

12. Anonymous says:

"but I think you have to consider that after #10 he's feeling confident and he is more focused and there may actually be a greater chance of #11 going in."

No, that's the point - there ISN'T a greater chance of #11 going in. You can talk all you want about immeasurable mental effects, but this one, if it existed, IS measurable, and the inability of several studies in several different settings to find correlation can only be seen as evidence that the effect either doesn't exist or is smaller than is measuable by these studies.

13. Loftur KristjÃ¡nsson says:

There is one thing about the play calling that you missed. There are some quarterbacks that have the authority to change plays at the line of scrimmage if they see a favorable matchup or a certain coverage. Wouldn't this make play calling for the defenses more difficult when playing against those kind of quarterbacks?

14. Anonymous says:

Regarding 'the hot hand' - my thoughts:

Having been a basketball player, there are definitely times when you get a hot hand, and there are definitely times when you hit a slump.

The problem with a statistical analysis is that the 'hot hand' is a minor change in the underlying probability distibution function, and is completely inaccessible to staticians. Typically one applies the ergodic theorem to assume that statistics taken over time represent the ensemble statistics, but this is not valid.

As an example, let's say a basketball player hits 75% of his freethrows every week, over a 10 year career. One week, he hits all 20 of them. It is simply not possible to say that this 'hot' week was a realization of the "hot" underlying probability function (with a mean of say 90%), or a realization of the constant probabilty function (with the mean of 75%).

-bob

15. Anonymous says:

"The true countermeasure" is to determine which teams apply this analysis, and predict what their defense will be based on the analysis (which the O will have as well), and then do the opposite.

that is, until they get enough data to know that you know they know, but they won't know that you know that they know that you know :)

-bob

16. Anonymous says:

Well, you sure are putting a lot of faith into those studies.

fans alike tend to believe that a player’s chance of hitting a shot are greater
following a hit than following a miss on the previous shot."

That statement is saying that the probability of making a shot after a hit isn't any higher than the probability of missing a shot after a hit. That's not what I'm talking about. What I'm talking about is a string of hits, followed by a hit. If a player makes five shots in a row, what is the probability of making the sixth?

And to make something clear - this is so not flipping coins. Making free throws isn't a 50/50 proposition. Free throws are skill, physics and luck.

What I'm saying is that if Larry Bird and I took 100 free throws each, he would make more and go on longer streaks than I would. I would attribute this to several things...he has more physical and mental training than I do when it comes to shooting free throws. His streaks would be longer because he is more physically and mentally balanced when it comes to free throws. Luck would have less impact on the outcome of the contest than say you and I shooting free throws.

17. Anonymous says:

The study referenced specifically states that the proportion of shots was found unrelated to the number of previous shots in a row missed or made. Also, your hypothetical Larry Bird situation is somewhat ridiculous - if he were so mentally trained, he'd be more likely to find his mental state that yielded optimal shooting results and keep that state rather than alternate between streaks. It's also disprovable - actual professional players' streaks have been studied and no players are more streaky - every shot is an independent event. I don't understand why people feel like statistics are all great and applicable until they contradict their own personal experiences: the whole point of statistics is to provide objective analysis that is MORE powerful than personal experience, not simply to confirm intuitive notions.

18. Anonymous says:

I'm the guy who first mentioned Larry above.

Someone with a bit more time go ahead and try this, setup a coin flip simulation in excel and assign a .886 probability to Heads and a .114 probability to tails. We will interpret Heads as a Make and Tails as a Miss (these are Bird's career FT make and miss percentages). Run the simulation 10,000 times. Take a look through the data, you will find some very long runs of Heads that are a result of nothing more than random variation. Looked at in any other context, however, (i.e. a basketball game) these runs will be mistaken for something with more meaning. We can even calculate the odds of a run of certain lengths appearing over this sample (it might surprise you to try). This mistake is likely to be made by the player (all you ex-ball players saying you've "felt" the hot-hand) as well as the spectator. My point about skill being a factor was that better players are going to have many and longer of these Heads runs and thus we are more apt to call this the hot-hand b/c it (due to their underlying probabilities from being good) is outside (or at least distant from) the normal expectation when compared to the entire league. So it doesn't make sense to compare Bird to me, you, or anyone else - of course he will make more, he has a higher underlying probability. You need to compare Bird to Bird which is what they did and they failed to find evidence of a hot-hand. Bird's Heads probability did not rise significantly after making one or more shots in succession....Did that make sense?

-Not Bob

19. Anonymous says:

Well then let's take this discussion out of the context of shooting free throws.

http://www.depts.ttu.edu/hs/hdfs3390/hh_brettandgwynn.htm

Certainly not concrete, but heck it came from Bill James.

20. Anonymous says:

And here is another article, from the same guy you referenced earlier.

http://www.depts.ttu.edu/hs/hdfs3390/hh_gilovich.htm

21. Anonymous says:

anonymous (Not Bob.)

Just because a sequence is a possible realization of a random process, does not prove that it is. And, more importantly, it does not indicate in any way if the probability distribution governing the random variable changes in time.

In my opinion, the famous study on hot hands in bball completely misses the point. A shot does not depend on the previous shot, that seems to be proven well. The hot hand is not inconsistent with each shot being a random variable and uncorrelated with other shots.

Their analysis of 'hot nights' is closer to the mark, but incomplete (7 players over 10 games).

One remarkable fact of that paper is how very few of the "runs" were what were expected (a couple had longer runs, most were shorter). Thus their model was invalid, and yet they use it to make their conclusions.

First of all, in a game, there are a ton of other direct factors involved. If you have the 'cold hand' well you don't shoot from outside as often, you will drive to the net to take a higher percentage shot, or you pass the ball more. The 'hot hand' takes riskier shots and shoots quicker - none of this was addressed in the paper.

Some examples:

- in his first or second year, michael jordan scored 63 points in a playoff game against the celtics - this is not a hot hand? It is just random? Then, i suppose anyone could do that - why doesn't dennis rodman ever have a 63 point game in the playoffs?

- Vinnie johnson of the detroit pistons scored 24 points in a row in the 4th quarter of a playoff game (also against the celtics iirc). Seriously, after vinnie got 22 points, wouldn't you give him the ball for the next shot?

- isiah thomas scored 16 points in the final 93 seconds of a game (down 9 ) to tie the game, is this not a hot hand?

Sure these are anecdotal, but the point is that hot hands are not just very short sequences that are correlated. They are even more relevantly great performances.

An analysis needs to be done based on much more than just the simple analysis done in that paper - it is simply incomplete.

-bob

22. Tarr says:

Really quickly on this hot hand debate:

Of course not any NBA player can score 63 in a game. But when you average 30 shots a game and 37+ points, like Jordan did that year, it's not as statistically unlikely.

Moreover, a lot of great "hot streak" sequences or games are not solely statistical variations, but they aren't the hot hand either. They simply reflect a change in the conditions in the game. Speaking from personal experience, if I'm playing basketball and I end up with a smaller/less athletic defender, I often end up getting a quick run of points. It's not that I suddenly became a better shooter or got a "hot hand", it's that I am getting easier shots from better places on the floor.

Speaking further from personal experience, there are days when I feel like I've got a hot hand, and a few shots in a row have fallen and I don't think I can miss. You know what? I miss the next shot plenty of the time.

23. Tarr says:

Getting back to the original post:

- I thought it was a great move that Shannahan called the same play for the TD and the 2-point conversion on consecutive plays on Sunday. Obviously you can't do that all the time, but if your play calling is truly random you will occasionally run the same play twice in a row.

- Sometime a couple years ago I heard a comment by some scout for some team stating that the Patriots play-calling showed no tendencies; i.e. that they were truly random in left/right/middle, run/pass breakdowns. Unfortunately I can't remember the details, but it wouldn't surprise me if Belichick's staff makes an effort to randomize play calling.

BB, are you ever going to put out team ratings this year?

24. Brian Burke says:

Tarr-Yes, probably beginning after week 3 or 4.

25. Anonymous says:

Bob, you make some good points that I agree with, others that I do not. Tarr's point about MJ sums some of that up. As humans I think it is very difficult to think of ourselves as random variables, and we may not be! But that's philosophy, I suppose. Either way, I enjoyed the discussion. See you out there.

26. Anonymous says:

Tarr Quote:{Of course not any NBA player can score 63 in a game. }

Anon quote: {Tarr's point about MJ sums some of that up.}

If it is truly a random variable, then we could take dennis rodman (ok bad example, but someone with a similar FG% to MJ), give him the 30 shots per game, and expect similar results. That doesn't happen.

By my point is more about the simple statistical definition of 'hothand', and how it is wrong. The PD that each realization (each single shot) is taken from is not stationary. It changes on all time scales, and the statistical analyses that I have seen (which admittedly is not very much) do not address that.

and i agree, good discussion. Lots of interesting stuff to chew over on this site.

-bob

27. Anonymous says:

I agree, unlike a free throw, which is taken from a stationary point, each shot in the game has many more varibles. Was it an off-balance shot? Was the shot taken against a good defender? The player with the "hot hand" seemingly hits a string of these harder, "lower probability" shots (MJ comes to mind).

28. Anonymous says:

Math Guy by day, Football Nut the rest of the time. Scouting and Self Scouting Team Tendencies are something coaches work 18 hour days because of.
It just has to be random "enough" to be effective.

29. Anonymous says:

it always boils down to the human factor

30. Anonymous says:

maybe Belicheck is not cheating, just figured out how to out coach himself.