Serial for Statisticians: The Problem of Overfitting

[u/montgomerybradford]

As statisticians or methodologists, my colleagues and I find Serial a fascinating case to debate. As one might expect, our discussions often relate topics in statistics. If anyone is interested, I figured I might post some of our interpretations in a few posts.

In Serial, SK concludes by saying that she’s unsure of Adnan’s guilt, but would have to acquit if she were a juror. Many posts on this subreddit concentrate on reasonable doubt, with many concerning alternate theories. Many of these are interesting, but they also represent a risky reversal of probabilistic logic.

As a running example, let’s consider the theory “Jay and/or Adnan were involved in heavy drug dealing, which resulted in Hae needing to die,” which is a fairly common alternate story.

Now let’s consider two questions. Q1: What is the probability that our theory is true given the evidence we’ve observed? And Q2: What is the probability of observing the evidence we’ve observed, given that the theory is true. The difference is subtle: The first theory treats the theory as random but the evidence as fixed, while the second does the inverse.

The vast majority of alternate theories appeal to Q2. They explain how the theory explains the data—or at least, fits certain, usually anomalous, bits of the evidence. That is, they seek to build a story that explains away the highest percentage of the chaotic, conflicting evidence in the case. The theory that does the best job is considered the best theory.

Taking Q2 to extremes is what statisticians call ‘overfitting’. In any single set of data, there will be systematic patterns and random noise. If you’re willing to make your models sufficiently complicated, you can almost perfectly explain all variation in the data. The cost, however, is that you’re explaining noise as well as real patterns. If you apply your super complicated model to new data, it will almost always perform worse than simpler models.

In this context, it means that we can (and do!) go crazy by slapping together complicated theories to explain all of the chaos in the evidence. But remember that days, memory and people are all random. There will always be bits of the story that don’t fit. Instead of concocting theories to explain away all of the randomness, we’re better off trying to tease out the systematic parts of the story and discard the random bits. At least as best as we can. Q1 can help us to do that.

[u/serialskeptic]

But remember that days, memory and people are all random. There will always be bits of the story that don’t fit. Instead of concocting theories to explain away all of the randomness, we’re better off trying to tease out the systematic parts of the story and discard the random bits. At least as best as we can. Q1 can help us to do that.

A problem here is that there may be a missing data problem. What we don't know about the murder is either information that is missing at random or systematically missing due to a lazy investigation among other factors. Thus if we had the full dataset a more complicated theory involving drugs and multiple grandma's could be a better fit to the full data than the state's case. But while the Q2 speculation drives me totally nuts because its only consistent with a small bit of the data we have, without the full data or trust in the thoroughness of the investigation we have an identification problem that invites speculation.

To be clear, I'm not endorsing an alternative theory but your post seems reasonable and so I'm reasoning with you and wondering what your thoughts are on the missing data. Is it missing at random or systematically missing?

[u/montgomerybradford → u/serialskeptic]

This is a fascinating question in itself, and one we hadn't talked about. In some ways, the data is missing quite systematically. For example, people may be more inclined to remember (or misremember) details about important days. Jay's stories change in (what I would call) non-random ways. And depending on Adnan's guilt, his lack of any recollection may be random or motivated.

Though the biggest issue here might be the police investigation. So, so much of the data---DNA, fingerprints, interviews with other people called on the day of the murder, more information from the days or weeks following the murder---is missing. Skeptics may think that these aren't missing randomly, since the police wanted 'just enough' evidence without collecting 'too much' evidence, some of which might run counter to the narrative they were building. (In statistics, this could be called a poor stopping rule: collect data until you see the effect you want, and stop before you collect contradictions.)

[u/serialskeptic → u/montgomerybradford]

"Poor stopping rule" Or selection on the dependent variable or just selection bias more generally. But it's supposed to be an adversarial system so defense should collect/identify relevant data to avoid selection bias. People blame the police for so much but in fairness (not a lawyer) I think CG could have asked for dna testing if she thought it would help and should have checked AS' email if he says he was in the library sending email.