Do winners wear red? Does the truth wear off?

Last week I had to go to three meetings in a row so I wore my red sweater. A non-sequitur, you say? Well, maybe, but I had a theory. Based on this article in Nature, I had decided that wearing red would enhance my performance in conflicts, which I pessimistically assumed would arise in these meetings. Later in the week, I had a meeting where I didn't at all get what I wanted and I attributed it to the blue (loser) sweater I wore that day. Then I heard from my friend Sui Sui, international business professor at Ryerson, that the Nature study had been debunked. In the 2008 Olympics it turned out that the blue-shirted athletes won more often. And in the 2004 Olympics (the source of the data used in the Nature paper), it now appeared that the shirt colour allocation was not really random.

This brought to mind a very interesting article I read last month in the New Yorker, titled "The Truth Wears Off" by Jonah Lehrer. The author describes a variety of effects that were first estimated as being quite strong but were subsequently shown to be weak or non-existent. He attributes the tendency of "decline" to three main forces.

1. Publication bias: In order to get published your paper needs to have strong, significant results.
2. Selective reporting: Scientists screen their own work, tossing out results that don't fit with their priors.
3. Study-level random effects. Although Lehrer doesn't really nail this point, I think what he is getting at is that the significance levels in studies are calculated as if every observation in the study is independent. But what if the "apparatus" used to conduct the multiple measurements has a problem? Then over and over again it will give similar results. If the flaw is random, it doesn't bias the result up or down but it does bias the significance levels so we think that we have something when we don't. In the case of the red-shirt study, there seems to have been something about the 2004 Olympics that tilted contest victories towards red shirts but it was not an enduring effect (i.e. not truly biological or cultural, say) so it didn't show up in the 2008 replication.

One of the letters to the editor points to a case where an estimate started out too low but was only gradually revised upwards.