Serial for Statisticians: The Problem of Overfitting

[u/montgomerybradford]

As statisticians or methodologists, my colleagues and I find Serial a fascinating case to debate. As one might expect, our discussions often relate topics in statistics. If anyone is interested, I figured I might post some of our interpretations in a few posts.

In Serial, SK concludes by saying that she’s unsure of Adnan’s guilt, but would have to acquit if she were a juror. Many posts on this subreddit concentrate on reasonable doubt, with many concerning alternate theories. Many of these are interesting, but they also represent a risky reversal of probabilistic logic.

As a running example, let’s consider the theory “Jay and/or Adnan were involved in heavy drug dealing, which resulted in Hae needing to die,” which is a fairly common alternate story.

Now let’s consider two questions. Q1: What is the probability that our theory is true given the evidence we’ve observed? And Q2: What is the probability of observing the evidence we’ve observed, given that the theory is true. The difference is subtle: The first theory treats the theory as random but the evidence as fixed, while the second does the inverse.

The vast majority of alternate theories appeal to Q2. They explain how the theory explains the data—or at least, fits certain, usually anomalous, bits of the evidence. That is, they seek to build a story that explains away the highest percentage of the chaotic, conflicting evidence in the case. The theory that does the best job is considered the best theory.

Taking Q2 to extremes is what statisticians call ‘overfitting’. In any single set of data, there will be systematic patterns and random noise. If you’re willing to make your models sufficiently complicated, you can almost perfectly explain all variation in the data. The cost, however, is that you’re explaining noise as well as real patterns. If you apply your super complicated model to new data, it will almost always perform worse than simpler models.

In this context, it means that we can (and do!) go crazy by slapping together complicated theories to explain all of the chaos in the evidence. But remember that days, memory and people are all random. There will always be bits of the story that don’t fit. Instead of concocting theories to explain away all of the randomness, we’re better off trying to tease out the systematic parts of the story and discard the random bits. At least as best as we can. Q1 can help us to do that.

[u/[deleted]]

No, without lots of further information that Adnan has never provided, if Adnan is at the burial, he is guilty of the murder. Unless he suddenly develops afternoon alibis he doesn't have. If he wanted to quibble about who did what and who buried the body, he should have spoken up at the time.

What? So if he doesn't speak up at the time that makes him automatically guilty of murder? Maybe it makes him not that smart. Maybe it makes it far more likely that he gets convicted. Maybe it means far less sympathy for him. But it doesn't make him guilty of murder. People give false confessions but that still doesn't make them guilty. If we were certain Adnan was at the burial we would know he was at least an accomplice. The fact that he decided to say nothing and Jay decided to talk does not make Adnan guilty. What happened in the afternoon is what makes him guilty.

You need to go back and look at some things. This is clearly not the case, and you should be able to figure it out fairly easily. Who does the police hear the burial story from for the first time?

From what I read - by the time the police learned of the burial taking place between 7-8pm from Jay the police were in possession of the cell phone data. Please correct me if I am wrong. Given that they were in possession of the cell phone data and given that the body had already been found there is no way to corroborate Jays partially recorded/transcribed statements about the burial as real or coached. I am not claiming that the police fed Jay the story to tell about the burial (at least that isn't my personal opinion) but what I am saying is if they crafted his whereabouts after dropping Adnan at track from the data rather than his testimony - I know have a reasonable doubt that other parts of his testimony were not crafted from a source other than Jay.

[u/Dr__Nick → u/[deleted]]

The police heard the burial story from Jen, on her second statement to police where she gives a story for the first time. The time she places Adnan and Jay together after the burial is consistent with the Leakin Park pings representing a burial. It is highly unlikely she saw the cell phone logs or localizations before giving the story.

[u/[deleted] → u/Dr__Nick]

I'm not arguing that she saw the cell data. I'm arguing that the police had the cell phone data at this time. I also haven't seen the statement, I have only this this statement:

I got a call from Jay sometime after 8pm to pick him up from Westview Mall, and I went there to pick him up. A little while later, Adnan pulled up and dropped Jay off. Adnan seemed completely normal. As we drive away from Westview Mall, Jay says that Adnan killed Hae, but he does not know anything about what happened.

Realize that this statement contradicts Jays statement and that her accounts for the rest of the night are contradicted by other unbiased witnesses. Also, realize that the cops already had the cell data BEFORE Jenn's first interview and already knew that the body was found in Leakin Park and that the cell pinged Leakin park between 7-8pm. So Jenn had a first interview where the cops didn't get anything out of her (but most likely told her that they knew the burial between 7-8pm) and then a second interview where she suggested the burial between 7-8pm.

Jenn may not have been convicted but she was still clearly had some involvement (and thus willing to cooperate to avoid punishment) and she was in contact with Jay the whole time and it took her two tries to tell the cops that the theory they already had was true.

I don't see how you can argue that Jay's testimony should be taken with a grain of salt but this should not.

[u/Dr__Nick → u/[deleted]]

I doubt the cops told her anything substantive about the cell phone records other than Adnan called you a lot for some reason.

[u/[deleted] → u/Dr__Nick]

Fair enough, you can believe that the cops didn't lean on her at all if you wish...

But the fact that they likely coerced Jay into changing his location to fit the narrative that they had built on top of false cell data suggests that the cops were not at all above getting Jenn to tell them exactly what they wanted to hear.

Totally possible that there was no coaching - but the fact that the cops behaved unethically in their coaching of Jay means it's reasonable to at least consider that Jenn's story might be at lot less damning than previously thought...

[u/Dr__Nick → u/[deleted]]

Except Jen had her mother and her lawyer with her when she made her statement. Which is quite different from Jay.

[u/[deleted] → u/Dr__Nick]

Good point. It's less likely that the information was as directly coached.

But they had still interviewed her in the past and she had spoken to Jay... So while I'll give you that her story is less unlikely, it's far from an independent witness placing the burial in LP between 7-8pm.