If you're going to be doing sports statistical research, this is required reading. Click to expand. (From xkcd.com)
Significance
published on 4/06/2011
Subscribe to:
Post Comments (Atom)
If you're going to be doing sports statistical research, this is required reading. Click to expand. (From xkcd.com)
For more in-depth discussion on similar matters, check out Andrew Gelman's slides on "Statistical Challenges in Estimating Small Effects": http://www.stat.columbia.edu/~cook/movabletype/archives/2011/04/my_talk_at_nort.html
The problem described in XKCD is a multiple comparisons problem (http://en.wikipedia.org/wiki/Multiple_comparisons). It has basically nothing to do with problems of estimating small effect sizes, except that they're both statistical pitfalls.
Don't forget the alt-text:
So, uh, we did the green study again and got no link. It was probably a-- RESEARCH CONFLICTED ON GREEN JELLY BEAN ACNE LINK; MORE STUDY RECOMMENDED!
Ups to DSMok1 for posting that; Taleb's stuff has really hammered home this for me too.
More or less, he didn't use a Bonferroni's correction for multiple comparisons.
I would guess that Gelman would say that effect sizes are rarely precisely zero, and that almost all testing problems can be analysed at least as well by treating them as estimation problems. According to this philosophy, the problem is to simultaneously estimate the effects of all the different jellybean colours, and in any sensible way of doing this it will be plausible that all the effect sizes are basically zero.
lol i love it.
EXACTLY why stats are worthless without common sense.
Tim, I think it's more of a case of "stats can be misleading if you don't understand how stats work". Although that's less sexy, I guess.
The real joke is that they're frequentists... they should join the 21st century and switch to Bayesian.