Friday, October 13, 2006

The Lancet Study

I've just been having a little look at the claims made in the Johns Hopkins/Lancet study on mortality rates in Iraq. Now my aptitude for maths is low, but the survey seem utterly ridiculous to me.

Take deaths by car bomb, for example. The survey sample of 300 violent deaths from March 2003 to July 2006 (page 7 of the survey) is extrapolated to approximately 600,000 for the country as a whole. Of the 300 violent deaths, 30 (10%) were the result of car bombs in the year June 2005-June 2006. Using the survey's methodology, I believe that equates to 60,000 people killed by car bombs in one year. The most recent data available on the Iraq Body Count website lists 15 car bombs in the first half of September (ignoring bombs which targeted non-Iraqi forces); taking the highest figure for reported deaths, these bombs killed 75 people. That’s an average of 5 people killed per car bomb. On that basis, 60,000 deaths would require 12,000 car bombs in one year, or 33 per day. Either that or there are hundreds of massive car bombs killing hundreds of people which are going totally unreported. I respectfully suggest that the study is a load of bollocks.

And what about the 5.5 deaths/1,000/year for pre-war mortality rates? Today's UK rate is 10.13 deaths/1,000. I seem to recall that before the war the likes of Galloway and Pilger kept telling us that hundreds of thousands of people were dying because of sanctions on Iraq. And yet now we're supposed to believe that the mortality rate was half that of the UK? Conclusion - they're all full of crap, the lot of them.

Natalie Solent has a link-rich post highlighting the political bias of those behind the study.

And here are Iraq the Model's thoughts.

13 Comments:

Anonymous JuliaM said...

"Now my aptitude for maths is low.."

So's mine. But my aptitude for poorly researched, contradictory, blatantly political bullshit is just fine.....

I particularly like the bit about how they don't rely on official figures because of the infrastructure breakdown, etc, then go on in the next breath to claim that most of the claims were backed up by death certificates.....

Talk about having your cake and eating it too!

5:59 pm  
Anonymous JohnM said...

And yet now we're supposed to believe that the mortality rate was half that of the UK?

I agree that given the Stoppers were claiming an absurdly high death toll due to sanctions. This puts them in the absurd position that if they are telling the truth now, they must have been exagerating before.

However, it is quite likely that the death toll in the developing world is lower than in the west for the simple reason they have a lot more children. Put simply, an aging population will naturally have a higher death rate than a young one.

Harry's place also has video of the impecably neutral (cough, cough) Richard Horton.

10:54 pm  
Blogger Laban said...

I think the mortality rate is a non-starter. It should be much lower because of the age profile of Iraq - there are many more elderly people in the UK.

The car-bomb point is good though.

11:00 am  
Blogger Lopakhin said...

It's true that the crude death rate figures are counterintuitive and a bit misleading. But even given that, 5.5/1000 is very low for pre-war Iraq. The World Health Organisation figures for 2001 give 9.0/1000 - see here. And, as a previous commenter mentioned, the death toll due to sanctions is still stressed, even by the Lancet study itself - for instance, see Appendix E of its companion document.

1:00 pm  
Anonymous dsquared said...

The notes to that table say that they think that quite a few cases where a mortar shell hit a car were misclassified as "car bomb", so we know that one shouldn't put too much stress on the detailed breakdown of the data.

But what's most likely happened is that although (accepting this for the sake of argument) the average car bomb kills 5 people, by chance this survey happened to pick a street where a particularly large one had been placed and killed 20.

So the figures are biased upward by this? No, because it's also very likely that equally by chance, it picked a few streets where nobody had lost a relative at all (in the first survey, they managed to find a cluster of zero deaths in the middle of Sadr City during massive insurgent fighting there). Of course, this wouldn't have been on the news either and would have biased the excess death estimates down.

This sort of thing is basically why god made confidence intervals. If you're going to use analysis of this kind to chew away at the central estimate, then you have to do it symmetrically and also look for numbers of deaths that seem unreasonably low. Personally I think that it's not a productive exercise to focus on the big number, and just to take the qualitative conclusion "things have got worse, and they have got a lot worse rather than a little bit worse".

1:59 pm  
Blogger Lopakhin said...

Oops, sorry - here's that companion document.

4:37 pm  
Blogger DFH said...

RE death rates - fair enough, the comparison with the UK is pretty much irrelevant, but I'm grateful to Lopakhin for pointing out that higher WHO figure.

Dsquared: “we know that one shouldn't put too much stress on the detailed breakdown of the data… Personally I think that it's not a productive exercise to focus on the big number, and just to take the qualitative conclusion "things have got worse, and they have got a lot worse rather than a little bit worse".

Never mind the facts, feel the qualitative conclusion. Priceless. Reminded me of that dickwad Foucault: “It's often the case that people say something that, at the factual level, isn't true, but which refers to another, deeper meaning, which cannot be assimilated in terms of precision and observation.”

7:30 pm  
Anonymous dsquared said...

Y'know, "dickwad", if you don't know what you're talking about and you're in conversation with someone who does, it's often a good idea to leave out the attitude and personal insults. By "qualitative conclusion" here, I am referring to a hypothesis test which can easily be made on the basis of the data.

In other words, the data clearly allow one to reject the null hypothesis that the invasion of Iraq had not affected the death rate with a very high degree of confidence. The data also clearly allow one to reject the null hypothesis that the death rate had risen by a factor of less than two, also with high degree of confidence.

These are the freaking facts, moron. They are true, at the factual level and all others. It is complete crap like your amateurish attempts to play "well it seems to me" with made up numbers about car bombs which are embarrassingly Foucauldian, and are both imprecise and clearly not based on observation. I wish I hadn't bothered being polite to you in the first place now.

"Priceless". If only you knew.

8:55 pm  
Blogger DFH said...

I wish I hadn't bothered being polite to you in the first place now.

That's the differnce between us - I intend to be as polite to you as I always have been.

Just out of interest, is that really the best picture you could come up with for your Comment is Free column? In all honesty, sweat-clagged ginger hair is not a good look.

Anyway - back to the survey. Bullshit

“there's something troubling about a survey that claims to have discovered hundreds of thousands of unreported deaths, by looking past official channels, yet that also claims that 80% of those deaths that it discovered have official death certificates”

1) 99% of residences had people at home
2) 98% of residences had heads of household (or spouses) at home and willing and able to provide the requested information
3) Thorough explanations and double-checking and proofs of information were possible in well under ½ hour per interview...

1:22 am  
Blogger Natalie said...

"I wish I hadn't bothered being polite to you in the first place now."

From the person who gave us this.

1:50 am  
Anonymous JuliaM said...

"These are the freaking facts, moron. They are true, at the factual level and all others."

Because I say so! Fear my towering intellect, mortals!

Blimey, what with this, and Richard North's pompous banning of Cranmer (in the thread above), there seems to be a lot of very 'precious' people on the net lately....

It's becoming a lot like a dressing room at RADA!

5:30 am  
Blogger Lopakhin said...

Iraq Body Count, in their critique of the new Lancet survey, pick up on the car bombs point too:

http://www.iraqbodycount.org/press/pr14/1.php

Lancet estimates 150 people to have died from car bombs alone, on average, every day during June 2005-June 2006. IBC's database of deadly car bomb incidents shows they kill 7-8 people on average. Lancet's estimate corresponds to about 20 car bombs per day, all but one or two of which fail to be reported by the media. Yet car bombs fall well within the earlier-mentioned category of incidents which average 6 unique reports on them.

'Baghdad-weighting' of media reports, even if applicable to car bombs, is unlikely to account for this level of under-reporting, as half of the car bombs IBC has recorded have been outside Baghdad. The Pentagon, which has every reason to highlight the lethality of car bombs to Iraqis, records, on average, two to three car-bombings per day throughout Iraq, including those hitting only its own forces or causing no casualties, for the period in question.

10:28 am  
Blogger Squander Two said...

> If you're going to use analysis of this kind to chew away at the central estimate, then you have to do it symmetrically and also look for numbers of deaths that seem unreasonably low.

I think that would be true for less extreme wrongness than we have here. But we're not talking about mild discrepancies here; we're talking about outright bollocks.

The claims in the Lancet report have real implications. Demonstrating that one of those implications is not just false but wildly, ridiculously far from being within spitting distance of truth is a valid criticism of those claims. I don't believe that it is a valid defense of the study to say that, ah, yes, but lots of their other claims could be just as wildly and demonstrably inaccurate and they probably all cancel each other out.

Firstly, I think the onus is on the report's authors to have done some common-sense checking themselves prior to publication -- they should have noticed and investigated and mentioned the car-bomb discrepancy themselves. That they did not is evidence of sloppiness or bias or both. I would never have submitted any sort of statistical report without checking it to see if it looked likely and realistic, and I would have expected to be hauled over the coals had I done so. And quite right too.

Secondly, the onus is on the report's authors to give evidence of why they think all their inaccuracies will tend to cancel each other out. "Because it's statistics and they always do" is not an acceptable answer.


> amateurish attempts to play "well it seems to me" with made up numbers

You know, I have to take my hat off to this sheer brazenness. Mr Squared, for those that didn't notice, is playing "Well it seems to me" at the professional level, and keeps his made-up numbers hidden behind English words where peopel tend not to notice them.

> But what's most likely happened ...

How likely, and according to what evidence? This is statistics; you can't just bandy a word like "likely" around. It means something. Give us a probability. 90%? 80%? Got a confidence interval?

> ... is that although ... the average car bomb kills 5 people, by chance ...

By approximately what chance?

> ... this survey happened to pick a street where a particularly large one had been placed and killed 20.

> So the figures are biased upward by this? No, because it's also very likely ...


Very likely? Exactly as likely as the "most likely" above? Or just likely to be roughly as likely? Come on.

> ... that equally by chance ...

Equally, eh? Any evidence for that equality? You see, I'd be willing to believe that your first use of "by chance" was just an English idiom and essentially meaningless if you didn't then go on to give us a second "by chance" which, you claim, is equal to the first one. OK, if you reckon these two chances are equal, what are they? 60%?

> ... it picked a few streets where nobody had lost a relative at all

Look, this is all true: these things are likely to have happened by chance. But a statistician's job is to say how likely and by what chance. And since, as Mr Squared has demonstrated, the answers to these questions make the difference between the report being very accurate and its being utter shite, it seems kind of odd that the authors appear to have made little effort to provide them.

6:22 pm  

Post a Comment

Links to this post:

Create a Link

<< Home