The recent post about the Verducci Effect and Let's Make A Deal didn't elicit the response I was looking for. Reactions ranged from denial to sadness to even anger. I think the game show story hurt more than helped my case of why I think the Verducci Effect is an illusion. And as I said in the original article, I'm not completely certain. But now, after developing my thoughts a little better, I'm more certain than before.
Game shows aside, I'll explain my thought process with an notional example, a mental exercise actually. I'm most interested in the injury aspect of the Verducci Effect, so I'll concentrate on that in this post. The injury rates I'm going to use are created only for clarity, and they are not intended to match the true rates. I'm also going to make some simplifying assumptions to illustrate the broader point. Please keep in mind this example is only intended to demonstrate a concept.
The Example
Assume every MLB pitcher's career lasts exactly 5 years. Also assume that the league-wide injury rate is 1 out of 5 years, defined as however you like--say being on the DL. Also assume that one year out of each pitcher's career can be identified, after the fact, as a "career year," which by definition assures us of two things: no significant injury and an upswing of innings.
Take 200 pitchers and randomly assign a number from 1 through 5 to each of their 5 respective years completely independently. When a 1 comes up, call that an injury year. So far, we've got a 1 in 5 (20%) injury rate across the league. Some pitchers will have multiple injury years, some won't have any, and they are completely independent.
In my head I'm thinking of a table of cards, 200 x 5. For now, the cards are turned up so we can see the numbers 1 through 5. 20% of the cards are 1s--injuries.
Now, take all the "career years" for the pitchers off the table by removing one card from each row, which by definition cannot be an injury year. Let's choose the highest card and remove it. What percentage of cards will now be 1s (injuries)? Before, there were 200 out of 1000 (20%), and now there are the same number of injuries (200) but fewer cards remaining (800). 25% of the remaining years are 1s (injuries). If we now selected a card at random, we'd have a 1 in 4 chance at finding an injury.
Shrink the Sample
Let's do the same exercise but with a sub-sample of 50 pitchers instead of 200. There are still 20% injury years, and if we take each pitcher's known "career year" off the table, 25% of the remaining years will be injury years. Turn over the remaining cards, so you can't see the numbers 1-5. Turn one card face up at random--what are the chances of finding a 1-card (an injury)? It has to be 25%. We started with 250 cards, but there are now 200 cards remaining and 50 of them are injury cards.
Shrink It Again
Repeat the exercise with 10 pitchers. Does this change anything? No. Originally 20% of the cards were injuries, and after removing the career year cards, 25% of those remaining are injuries.
Down to One Player
Now consider a sub-sample of just a single pitcher. Again, turn over all the cards so we can't see the numbers 1-5. There are originally 5 cards on the table, with a probability of 1 in 5 being an injury card. Take away one card, which we know after the fact is not an injury card, leaving 4 cards. Turn one card over--the card dealt immediately following the "career" year. What is the chance it's an injury?
It would be 25%. We had a league-wide injury rate of 20%, but following career/high-inning years we would retrospectively observe a rate of 25%. Even though the injuries were distributed completely at random and completely independently, we'd see a false connection between high-inning years and injuries in subsequent years.
Even a small difference would appear statistically significant with a large data set, but it would be an illusion. The original probability of a year being an injury year was always 20%, but after looking back and removing a year in which we're virtually assured of no injury, we'd see a 25% injury rate.
Try It Yourself
If you don't believe me, you can play the game yourself. Shuffle a deck of cards and deal out 4 face up in row. Those 4 cards represent 4 years of a pitcher's career. Every time we see a diamond card, we'll call that an injury year. We'll say the highest non-diamond card is a career/high-inning year. It goes without saying that, on average, 1 in every 4 cards will be a diamond--an injury. That's our true baseline rate.
After dealing the 4 cards, remove the highest non-diamond card and set it aside. Look at the card immediately to the right of the one you removed. What is the probability it is a diamond? If you said 1 in 4 you'd be mistaken. It's 1 in 3. This is the same illusion.
Try it. I did, 78 times and got a diamond on 26 tries--exactly one third (p=0.04 for the sticklers out there). You have to re-shuffle each time for it to be completely random and independent. Also, if the high "career" card is the right-most card, you can either throw out that iteration or loop around to look at the first card. The effect is the same. In fact, just look at how many of the 3 remaining cards are diamonds, and you'll eventually see that it's 1 in 3.
I'm sure there is a name for this, but I don't know it. If anyone is familiar, fill me in. Otherwise, I'm sticking with the Monty Hall Effect. Also, the Verducci Effect may still be real, but it would have to be shown that the observed injury rates significantly exceed the rate predicted by the effect of the illusion.
Maybe I'm wrong, and that's ok. But every time I deal 4 cards and remove 1 non-diamond, I keep seeing diamonds 33% of the time. Just like the Monty Hall game, you probably won't believe it until you try it yourself.
[Edit: I am now completely certain the paradox/illusion exists as I described. However, after a good discussion with commenter Vince (see below), I'm no longer convinced that the way I set up the question applies to how the Verducci Effect is truly applied. In other words, the illusion is real if you set it up the way I did. It's just that strange, and subtle differences exist in how you look at the problem. For example, if in the pitcher example, you removed the first instance of a non-1, you would see a 1 in the following block 20% of the time, just like you'd expect. But if you select the highest number in the row, or if you remove the final non-1, you'd detect a 1 next 25% of the time.]
Verducci Follow-Up
By
Brian Burke
published on 8/02/2009
in
other sports,
research,
reviews
Subscribe to:
Post Comments (Atom)
Maybe I'm not understanding this game correctly, but the card immediately following an 'injury' has less than a 20% chance to be an injury card itself. This would seem to indicate that in this simulation, an injury is more likely if the pitcher was healthy last season rather than injured.
I agree that the Verducci Effect is basically a combination of normal injury rates and regression to the mean, but this doesn't seem like the best way to illustrate it.
James-Here's an example:
H=hearts C=clover S=spades Cl=clubs
Shuffle, deal 4 cards:
4H AC 2H 3D
Remove the highest non-diamond (non-injury):
4H -- 2H 3D
Look at the card to right of the 'hole' created by the "career year." You'd see a diamond 1 in 3 times. (Or just count how many diamonds remain 0,1,2,or 3 out of 3. In the long run it will converge to 1 out of 3.)
I understand now. You nailed it.
This is a classic gaming theory exercise. He's right, its just counterintuitive.
You're choosing which card to pick out based in part on what comes after it, which does not match how people look at the Verducci effect. What if, instead of picking out the highest non-diamond, you picked out the first non-diamond face card? Then I think that the probability of having a diamond on the next card would still be 25% (or 13/51, if you're dealing the 4 cards without replacement from a single deck, making them non-independent).
Vince-Removing any non-diamond card would make it 1 in 3. It doesn't have to be the highest, first, last, or whatever--just not a diamond.
But I think your way of looking at it is similar to the way the Verducci analysis works in real life. Maybe, to be as realistic as possible, you'd remove the first non-diamond over 10 or something. But still, it doesn't matter as long as you take one non-diamond out of the mix. It's going to be 1 in 3.
The cards prior to the first non-diamond face card will be disproportionately diamonds (around a third of them will be diamonds - I think the proportion should be 13/40 with this setup, since they can't include any of the 12 non-diamond face cards), but the card following the face card will not be any more likely to be a diamond - it will be a diamond about 1/4 of the time (technically 13/51 since the cards in the simulation aren't entirely independent).
If you want to make the cards completely independent, you could take each of the 4 cards from a different deck. Take one card from each deck until you get to a non-diamond face card, then flip a card from the next deck and see if it is a diamond. It will be a diamond one fourth of the time. (If none of the first 3 cards is a non-diamond face card, then you have to throw out that iteration and move on to the next pitcher's first four years.)
Verducci is saying that pitchers with increased workloads often get hurt the next season
You seem to be saying that in a sample of X seasons, if you remove the season with the most IP, the injury rate will be higher than normal in the remaining seasons.
The two statements hardly seem related. You requier knowledge of the entire sample before manipulating the sample. Verducci is making predictions about the future.
You would need to shuffle the deck every time you look at a card for your game to accurately model the Verducci effect
Anon-I disagree. The Verducci analysis is looking back in the past to find a relationship, which naturally occurs due to post hoc selection bias. The analysis infers causation where non exists.
Vince/Anon-I see your point about shuffling the deck for every card, but let's be real. That's not going to make a practical difference.
PS The card game just illustrates the concept. Look at the math in the pitcher-injury example:
--200 pitchers, 5 seasons each
--1000 "pitcher-seasons"
--1 out of 5 is randomly an injury year (independently, with replacement)
--200 injury years in the sample
--Remove a known non-injury year from each pitcher, 800 pitcher seasons remain
--The chance of randomly finding an injury year is now 200 out of 800, or 1 in 4.
Why are you removing the diamond cards? Why can't the injury rate be 1/5 the next season with a 1/5 chance of a career year again?
Alternatively, lay out the entire deck of cards on the table.
--13 rows of 4 cards
--52 cards
--13 cards are diamonds (1 in 4)
--Remove 1 non-diamond from each row
--39 total cards remain
--13 diamonds remain
--randomly selecting any card gives you a diamond 13 out of 39 times = 1/3
In any statistical analysis there is a finite sample. There is a finite number of pitchers, seasons, and injuries. When Verducci/Carroll are looking through their data sample, they are essentially dealing with large but finite deck of cards.
Anon-That is how it works. In the example, any season has a 1 in 5 chance of being an injury, and a 4 in 5 chance of being a non-injury year.
You could have more than 1 "career"-type years, in which case removing additional known non-years them would enhance the illusion. The way I conceive of it is that when Verducci looks at a year-pair, at least one must be a non-injury year, removing it from the equation.
Brian - It seems like you're explaining why the effect exists in the sample of 1999-2005 pitcher seasons (or whatever)
I think the people disagreeing with you are thinking that you are explaining why some pitcher who has a Verducci Season this year will be extra likely to get hurt next year. Which you don't seem to be arguing
If you have a 1/5 chance of an injury season, doesn't that number remain constant regardless if you have an injury year or a career year. Your risk for injury is still 1/5 the next season no matter what you do.
The chance of randomly finding an injury year is now 200 out of 800, or 1 in 4.
But you're not randomly selecting a year - you're looking at the year immediately after the year that you removed (and you didn't remove a year at random - you removed the first breakout year). As I said in my last post, in your model the years prior to a breakout year are more likely than average to be injury years (since we know they weren't good years - they must be either injury years or lousy years), but the years after a breakout year only have injuries at the average rate (1/5, in this case).
Here's a simpler way to do it with cards, instead of dealing with face cards. Diamonds are injury years, spades are good years, and clubs & hearts are lousy (injury-free) years. In each row of cards, find the first spade. 1/3 of the cards prior to the first spade will be diamonds (1/3 clubs, 1/3 hearts, 0 spades). But the first card after the first spade (or any of the other cards after the first spade) will be equally likely to be a diamond, spade, club, or heart.
Vince and Anon above- I shouldn't have said "let's be real..." That sounds flippant. You are correct, period. I should have said: the non-replacement effect is extremely small at the beginning of the deck.
Vince-That's true. Every single card has a 1 in 4 chance of being a certain suit. As you deal cards forwards in time (with replacement), that will always be the case.
But if you deal all 4 cards, and then look them over removing 1 non-diamond, the remaining cards will be a diamond 1 out of 3 times.
That's the paradox. It all depends on which direction in time we are looking.
Just from a brief examination of the past two articles, what you are explaining sounds very similar to the Hurst Exponent. Hurst actually discovered this phenoma in the same manner that you are experimenting with the card deck. Not sure how it exactly relates but will look into it more and update further.
Vince-Your recent example above is good. I think what you're saying is "here's the right way to model pitching injuries." And I agree. Your model predicts no Verducci Effect.
What I'm saying is that Verducci/Carroll are not modeling it that way. They're modeling it the way I am with my card game and finding an effect.
But if you deal all 4 cards, and then look them over removing 1 non-diamond, the remaining cards will be a diamond 1 out of 3 times.
It depends on which cards you look over, and on how you decided which card to remove. If you remove a non-diamond and then look over every remaining card in the row, then 1/3 will be diamonds. But if you remove the leftmost non-diamond in the row (or the leftmost spade), and then you only look over the cards to the right of the card that you removed, then 1/4 will be diamonds. In general, if you decide which card to remove based only on that card and the cards to its left, and then you only look over cards to its right, then 1/4 of the cards that you look over will be diamonds. That has to happen because of independence.
And that's the way the Verducci analysis works. They pick out the "high workload year" based solely on that year and the previous seasons, and then they look at what happens in the following year. Their analysis matches my model, not yours. And they still find an effect, so it must have some explanation besides the one that you're giving.
Vince-I disagree. Try it yourself. The cards to the right will be 1/3 diamonds, and the cards to the left will be 1/3 diamonds.
I'll also disagree about what the Verducci methodology is. They must have had a finite data set. Your model waits for the future to happen, in which case it's a 1 in 4 chance. Their model excludes 1 card from the denominator in every case, after seeing all the cards, which is my model.
Vince-I disagree. Try it yourself. The cards to the right will be 1/3 diamonds, and the cards to the left will be 1/3 diamonds.
That's impossible (unless you're using some other procedure or you're getting small sample size randomness). If you pick out the leftmost spade from the row, then the cards to its right will contain identical proportions of diamonds, hearts, and clubs (since nothing distinguishes those 3 suits), and some of the cards will be spades, so there must be less than 1/3 diamonds.
I don't mean to be argumentative. I am enjoying our discussion and I respect your thoughts, or I wouldn't be responding. Too bad we aren't sitting in front of a white board.
But seriously try it! I did it again, this time with an Excel spreadsheet (so no shuffling or replacement issues, and pure independence).
I filled in each cell with a random number 1 through 5. I called the highest number in the row the break-out year, and looked at the cell immediately to the right. According to your reasoning I should see a one there 20% of the time. According to my reasoning I should see a one there 25% of the time.
For a sample of 200 rows, I found 55 ones to the right of the 'breakout' year. That's 27.5%, significantly different than the 20% you would expect (p=0.008 for n=200).
I'm also enjoying the argument - that's a main reason why I'm continuing it (the other main reason is that I'm pretty sure that you're wrong about this). The problem with your latest simulation is that you're picking out the highest number in the whole row, which uses information about what comes afterwards. As I've mentioned in earlier comments, you have to pick out a cell (or a year or a card) based solely on what's come before it - then independence will guarantee that whatever comes after it will match the base rate. That's how the Verducci analysis works - it picks out the first year in which your workload is substantially higher than its been before, without taking into account what you do in the following years, and then looks at your next season.
What if you call the first cell that is at least a 3 the "break-out year" and look at the cell to its right? Then you'll see a one there 20% of the time.
I think you might be right. That's the difference. If you pick out the first 3 or higher, I think you would see 20%. But if you pick out the last 3 or higher, or the highest number, you'd get 25%. I coded a couple simulations, and that's what I'm seeing.
The question is, which is closer to reality when analyzing pitcher injuries.
There's also a slight discrepancy in your method. It's not perfectly 25%, it's 24.967..% or 24.992% depending on how you handle the case of five injury years in a five year career.
There are 3,125 different career possibilities, In 1024 of them, no injury. In 1280 of them one bad year, which follows the best year in 320 of those cases. In 640 of them, two bad years, and 320 of those cases have a bad year after the best year. Another 160 cases have 3 bad years, with 120 of those showing the effect. 20 years have 4 bad years and all 20 of those have a bad year after the best year. Finally, 1 case has five bad years. I'm not sure what to select for the best year in that case. We can either throw the case out (24.967..%) or count it as a bad year after the best year (24.992%).
If the five bad year case is ignored, there are 3124 different cases, with 780 injury years after the best year. This gives the 24.967..% rate. Counting the five bad year case as a bad year after the best year gives 781 positives out of 3125 cases, or exactly 24.992%. Experimentally both are hard to distinguish from 25%.
Also, Vince has it right on the independence. If you're taking the best year considering all years, you've thrown away independence after that selected year by your selection method and so the impact you're describing does happen. If you select the critical year looking only at that year and the years before, then the following years are independent.
The only way to tell which Verducci used is to read the research if it's been published in sufficient detail. Anyone have links to the original articles as I'm somewhat curious as it's a story about having to be careful about your sampling, and I'd like to get the punchline correct when I tell the story.
I don't know the Verducci effect research very well, but I would think that they decide whether a pitcher is at risk for the effect in a given year based solely on previous years, which would mean that your model doesn't fit. That's what's implied by the definition that you linked to in your previous post (pitchers tend to underperform the year after they've had a substantial increase in innings pitched). And every year there are warnings about which pitchers could be in trouble this year because of the Verducci effect, and those warnings have to be based only on previous seasons.