Update 9:30pm Eastern. Shannon Love at the Chicago Boyz blog called foul on the Lancet 2004 study early on and, with vindication, reacts to David Kane’s new analysis of the 2004 Lancet Iraq death toll study: “Kane shows that if the Falluja cluster is included in the statistical calculations, the confidence interval dips below zero, which is a big no-no. Since the study’s raw data remain a closely guarded secret, Kane cannot be absolutely certain that the inclusion of the Falluja cluster renders the study mathematically invalid…but that’s the way to bet. In science, replication is the iron test. I find it revealing that no other source or study has come close to replicating the original study. All my original points still stand. Ah, vindication is sweet.”
Via Cox & Forkum, a fitting cartoon flashback:
One of the most useful roles of the blogosphere is its service as an open-source intelligence-gathering medium. You can draw on the expertise of people around the world at the touch of a button. We saw this with typography experts during the Rathergate scandal; Photoshop experts during the Reutersgate debacle; and military experts during the Jesse Macbeth unmasking.
Now, it’s the statisticians and math geeks’ turn. Remember that massively-publicized 2004 Lancet Iraq death toll study? It was cited in nearly 100 scholarly journals and reported by news outlets around the world. “100,000 Civilian Deaths Estimated in Iraq” blared the Washington Post in a typical headline.
There were attempts made by lay journalists to debunk the 2004 study (as well as the 2006 follow-up study that purported to back up the first). But none of those dissections comes close to a damning new statistical analysis of the 2004 study authored by David Kane, Institute Fellow at the Institute for Quantitative Social Science at Harvard University. I read of Kane’s new paper at this science blog and e-mailed him for permission to reprint his analysis in its entirety here so that a wider blog readership could have a look. He has given me his permission and adds that he welcomes comments and feedback. He’ll be presenting the paper at the Joint Statistical Meetings in Salt Lake City on Monday — the largest conference of statisticians in North America.
Much of the math here is mind-numbingly complicated, but Kane’s bottom line is simple: the Lancet authors “cannot reject the null hypothesis that mortality in Iraq is unchanged.” Translation: according to Kane, the confidence interval for the Lancet authors’ main finding is wrong. Had the authors calculated the confidence interval correctly, Kane asserts that they would have failed to identify a statistically significant increase in risk of death in Iraq, let alone the widely-reported 98,000 excess civilian deaths.
An interesting side note: as Kane observes in his paper, the Lancet authors “refuse to provide anyone with the underlying data (or even a precise description of the actual methodology).” The researchers did release some high-level summary data in highly aggregated form (see here), but they released neither the detailed interviewee-level data nor the programming code that would be necessary to replicate their results.
Kane has sent his paper to Lancet. But the blogosphere need not wait for Lancet to complete its review. If you’ve got a statistics background or know someone who does, have a look. Kane’s e-mail address is dkane-at-iq.harvard.edu. He’s a blogger himself at EphBlog.