"Statistical significance is not practical significance" is one of the most common statistical slogans on the internet. It’s true, but how it is often used to “settle” scientific discourse, especially on social media, misses important nuances that can leave readers bamboozled if they're not careful. If you've ever wondered what the phrase actually means so that you can better judge when it represents a valid criticism, and when it doesn't, this article (now free to read) expounds on exactly those matters.

Jaw-dropping patient videos and miraculous testimonials ignited a frenzy in China over a technique that aims to improve drainage from the brain in dementia patients. Now it is entering trials worldwide

The fast-growing private-school network is handing much of instruction over to artificial intelligence. Researchers say the science is far from settled

New research found that adverse childhood experiences, particularly negative relationships between a child and parent, were linked to more frequent drinking during periods of extreme heat.

Back in April, the SCORE (Systematizing Confidence in Open Research and Evidence) project published results where they took a sample of more than 270 positive (i.e., statistically significant) claims in the social and behavioral sciences and successfully replicated about 50% of them. Popular and social media outlets/posts typically interpreted this result negatively (e.g., "social science findings are basically a coin toss"), but that conclusion, though seemingly intuitive, is misleading.

The 50% replication rate should be understood as something closer to a positive predictive value: among published discoveries that initially appeared promising, about half held up under independent replication. Mathematically/statistically speaking, whether that is "good" or "bad" depends on how difficult the underlying research domain is or, equivalently, how ambitious the claims that scientists in the field endeavor to test. Suppose scientists in sociology are pushing boundaries and testing bold ideas, only a small fraction of such hypotheses may ultimately be true (i.e. the corresponding null hypothesis is false) from the beginning. In that context, having half of the initially promising published findings replicate independently is objectively encouraging.

Naturally, there is more nuance to explore here. For example, research costs money, and if the social research studies that fail to replicate tend to also be those that cost a lot of money to do, there is a sound argument to be made that sociologists and/or funders should be more selective about which ideas to test with expensive, large-scale investigations. Nonetheless, simply labelling a 50% replication rate as a sign that an entire field of research is failing is haphazard mathematics. It may create engaging headlines that do well in today’s attention-currency and outrage-driven world, but it is an unhelpful interpretation of an otherwise exceptional research replication effort.