03 July 2014

Cot death, Sally Clark and the prosecutor's fallacy

Sally Clark was an English solicitor who was prosecuted for the murder of two of her infant children. Her defence was that the deaths were both “cot death”, also known as “Sudden Infant Death Syndrome” or SIDS.

She was initially convicted. Much weight was given to the evidence of a medical expert, Sir Roy Meadow. He was a paediatrician and an expert in child abuse. He wasn’t a statistician. Using statistics from reports about SIDS, he estimated the chance of a cot death was 1:8543, and therefore the chance of a second was about 1:73,000,000. He arrived at this latter figure by multiplying 8543 by itself, squaring it.

Her conviction was upheld at a first appeal. However, at a second appeal she was acquitted, not because the statistics and calculations were incorrect (which they were), but because there was evidence of bacterial infection in the second child, information which was previously withheld from the defence. Sadly, the story does not have a happy ending; Sally Clark was found dead from acute alcoholic poisoning about four years after her acquittal.

I followed the first trial in the newspapers. I remember being distinctly disturbed by what I read; an intuition if you like, that there was something wrong, even though I understood that intuition is often a very poor guide to the results of mathematical calculations. It has taken me to now to understand what was going on.

I was reminded of the case of Sally Clark by reading Math on Trial by the mother and daughter Leila Schneps and Coralie Colmez. They are both mathematicians, and their book concentrates on the uses and abuses of maths in court cases. As such, it concentrates on the mathematics used in trials; the mathematics are often statistics and probability, and at times I found it really hard going.

SIDS


SIDS is a description, not a cause. It is applied to cases of natural death in infants, where no definite cause has been found. The causes of SIDS are usually felt to be a combination of genetic and environmental factors, even if these are either unknown or poorly understood. It hardly needs saying, that even if the cause or causes or SIDS are not known, this does not mean that there are no causes.

At the time of her first trial there were known associations with SIDS, such as smoking, maternal age and unemployment. As ever, an association is nothing more than that, it is not causation. Further, it was found that SIDS was rare in Hong Kong; the explanation was that there infants sleep on their backs, rather than face down. The subsequent “Back to Sleep” is said to have reduced the SIDS rate by about 50%. Again, the sleeping position is an association, not a causation. (The mechanism may relate to breathing patterns and brain wave activity which differ depending on the infants’ sleeping position.)

Murder is the unlawful killing of a human with malice aforethought (that is, intent). Usually, the prosecution must actively show and prove the means, method and opportunity of the perpetrator.

The Trial


Essentially, the prosecution argument was: either the deaths were due to SIDS, or murder. We will show, they implied, that SIDS is so mathematically unlikely, that the infants must have been murdered. That is, we will prove her guilt, not by proving her guilt, but by proving that she could not be innocent. This is a reversal of the usual ‘innocent until proven guilty’ rule. This distinction seems not to have been apparent at the time.

Professor Sir Roy Meadow was a paediatrician, famous for his descriptions of Munchausen’s Syndrome by Proxy, and child abuse. The original Munchausen’s Syndrome was described by Richard Asher; he found that there were patients who would self-harm to gain attention and satisfy their inner desires. Sir Roy extended this to parents (usually mothers) who harmed their children to satisfy their (maternal) needs. In child abuse, he is said to have coined “Meadow’s Law”, which is that “one death is a tragedy, two are suspicious, and three are murder”. This is almost a restatement of the quotation attributed to Joseph Stalin, that “one death is a tragedy, a million is a statistic”.

Sir Roy produced the 1:8543 incidence from a report; he did not accept the rather different figures from a similar report. This incidence relates to people in Sally Clark’s socio-economic group, and who aren’t smokers, unemployed or below 27 years of age. Boys are also more likely to die of SIDS than girls, compounding the overall possibilities. The real overall incidence of SIDS at that time seems to be 1:1300. “Seems” because the Back to Sleep campaign may have had an influence, and because if a registered cause of death is given as SIDS, and later information indicates a different diagnosis, the Registrar-General’s records will be changed. A subsequent search of the records may therefore not be complete, and reports and statistics on SIDS are often compiled retrospectively.

Sir Roy squared the 1:8543 figure to get 1;73,000,000. By itself this arithmetic is correct. But the (hidden) assumption is that two cot deaths are entirely unrelated, and this is incorrect. They are neither independent or “non-conditional”,  rather they are two events in which common factors (genetic and environmental) cannot be excluded Technically, two such events are described as a conditional probability. The mathematics underlying this is really very difficult for a non-expert.

Neither Sir Roy or the prosecution produced any statistics about double murders of infants.  The correct use of statistics in this case would be a comparison of the probabilities of double murder and the probabilities of double SIDS, not the presentation of only one “side” of the argument. There are other possibilities, such as one murder and one SIDS.

This misuse of statistics is known as the prosecutor’s fallacy, and is an example of Berkson’s paradox. This paradox arises when conditional probability is mistaken for unconditional probability.

Determining how often a second infant dies of SIDS is not easy. The best evidence I can find suggests that there is a dependency of between 5 and 10 in the case of a second death; that is, there is such an increased chance of a second death from SIDS. Intuitively, I would expect that a second death would be more common after a first, even if I could not quantify it. I would base this intuition on similar genetic and environmental conditions. Further, the possibility of a second murder does seem to be much less common after a first, and much less common than a double SIDS. In part, this may be because the murderer is likely to be imprisoned.

The figures of “5” and “10” have been produced from data relating to the incidence of SIDS. They have not been produced by “pure” mathematics, that is where there is a known correlation between two things, this has been factored in.

Evidence of the frequency of double murder was not given at the first trial; if it had, and a proper comparison of the rates of double murder and SIDS had been made available, it is very unlikely that she would have been convicted on statistical evidence alone. The medical evidence was not conclusive, showing no definite evidence of abuse or the lack of abuse. The infection that the second infant had was not disclosed; ultimately this proved vital.

If you are like me, you are probably finding all this statistical stuff confusing. Let us go through the logic and statistics step-by-step. I have rounded the figures to make the calculations more obvious.

The 1:8543 ratio (hereafter 1:8000) refers to a subset of people whose babies had died from SIDS. About 400,000 babies are born in the UK every year. Thus, if we take 1:8000 and divide this into 400,000 we find that 50 babies die of SIDS in a year. But, it is known that around 200 babies die of SIDS annually. We can only use the figure of 1:8000 if we know what subset of 400,000 it refers to—and we don’t. Overall, 200 babies in a population of 400,000 is 1:2000. So, you might say that Sally Clark’s subset had only a quarter of cases of SIDS compared to the general population. This isn’t relevant.

Around 20 to 30 infants were known to have been murdered annually. This represents 1:20,000 to (about) 1:13300. Thus, an infant is more likely to die from SIDS than to be murdered by a factor between 7 and 10.

Sir Roy squared 1:8000 to get 1:64,000,000 (actually 1:8543 to get 1:73 million). If we square 1:2000 we get 4,000,000. Likewise from 1:20,000 we get 1:400,000,000; and from 1:13300 we get about 177,000,000. Now, which is the most likely scenario?

So, even if we treat the two events as totally independent, which they aren’t, we can still show that SIDS is more likely than murder. We should realise that the events aren’t independent, but we cannot anyway easily quantify this. We might say that the dependency between two cases of SIDS and two murders are likely to be similar; but this is an unsupported supposition.


Repercussions


This picture, via Reuters, shows Sally Clark at the time of her release from prison; it speaks for itself:


Sir Roy Meadow was later struck off the General Medical Register, but reinstated on appeal.

The pathologist who performed the post mortem on the second infant was found guilty of serious professional misconduct by the General Medical Council.

The rules for expert witnesses were made more explicit; the expert was to provide a report for the benefit of the court; no matter whether instructed by the prosecution or the defence, the report should be unbiased. Expert witnesses were not expected to go further than their area of expertise. The evidence of an expert witness alone was no longer sufficient for a successful prosecution.

The cases of other women who had been convicted of murder were investigated; two had the conviction released, a third was acquitted.

DNA databases have been used as a “cold” trawl, trying to find matches. If there is no other evidence, this again is a prosecutor’s fallacy. If the evidence is confirmatory, the confirmation is likely to be correct. Prosecutors have been made aware of this problem, and to treat conclusions from DNA trawls cautiously.

And I, who was so disturbed by the initial conviction, did little about it until now.

Finally


In this instance, the court clearly didn’t fully understand why the statistics were wrong. The judge did warn the jury to use them with caution; understandably, the apparent conclusiveness of 1:73,000,000 must have played a large part in their determinations. It was a majority verdict, with two jurors not convinced.

Still, statistics and their misuse isn’t an easy subject. While advocates pride themselves on their ability to master a brief in a very short time, it cannot be possible for anyone to attain the level of knowledge and expertise that an expert has gained over years of study and work. Further, advocates and courts seem to have problems with “science” as a discipline. Either they seem to be overawed by it, or they find the ramifications to be so complicated that they ignore them as being not relevant.

The common law system is adversarial, that is, one side pits its strengths against the other. You can almost see it as a descendant of a trial by strength, or a joust. And at the end, despite the statue of Justicia and the designation, courts in this system do not dispense justice, they give law. The Code Napoleon is used as the basis of the legal system in many European countries; it is investigative, and certainly seems to be designed to discover the “truth”. As such, it might be regarded as superior to common law, but it’s unclear if it is actually any more accurate.

There’s another problem with courts. Giving evidence is straightforward, but cross examination is more difficult. To be a successful witness requires the ability to think rapidly to the questions that are put. It’s often said that advocates in court only ask questions to which they already know the answers, or think they know the answers. Yet, as their understanding is likely to be incomplete, their questions can be “para-relevant”, ones that cannot be easily answered in a sentence, rather they require considerable explanation. The perceived need for rapid answers is potentially dangerous; while “fast” thinking is useful for everyday problems, ones that we have previously encountered, “slow” thinking is much more appropriate to difficult problems, including ones that we haven’t previously met. But courts like certainty; a witness who is direct and who can answer questions without needing to think about them is likely to be favoured over one who is hesitant, who seems unable to provide a “yes or no” answer, and who needs time to think.

Some evidence was withheld at the first trial. There are rules about what is allowable and what is not allowable at trials. But the active withholding of evidence is not one of them. It is arguable that the pathologist who found bacterial infection, but didn’t reveal it, exhibited confirmation bias. Confirmation bias occurs when the conclusion is predetermined, and (subsequent) findings contrary to the conclusion are ignored. Confirmation bias is one reason for cross-examination.

Judges are supposed to be able to weigh evidence, and to base their conclusions on this. They also provide a summary at the end of a jury case, indicating, inter alia, areas of greater or lesser certainty; but what if they don’t really understand the evidence? Is a summary or judgement based on incomplete understanding of evidence which may be incomplete really valuable or reliable? Are judges able to think logically; above all, do they see faults in logic which bias their findings?


Resources and references


For a fuller account of Sally Clark’s trial, see the Wikipedia page here.

Math on Trial is available from amazon, here.

Both these sources indicate the cultural background at the time of the first trial.

For considerably more detail on the mathematics and statistics, see Dawid’s opinion, here; and Hill’s calculation of the probability of two cot deaths, here.

Wikipedia has details of the prosecutor’s fallacy here, and details of Berkson’s paradox here.

NB: be aware that these mathematical positions are very technical in parts, and are not easy to understand. I found it equally difficult to render their exactitude into common language.

Thinking, Fast and Slow, by Daniel Kahneman is also available from amazon, here.

Members of math.stackexchange.com helped with the mathematics, and supplied references.

Finally


As a former medical professional, I understand the medical issues here, even if my memory of statistics is rusty. As a parent of a daughter who died from cot death, I’m not an unbiased observer.



No comments:

Post a Comment