The Replication Crisis

In 2015, a team of 270 researchers tried to repeat 100 published psychology experiments. Fewer than four in ten reproduced the original result. This wasn't a story about fraud. Almost every original study was run by honest scientists using accepted methods. It was a story about how ordinary, well-meaning flexibility in data analysis, combined with a publishing system that rewards surprising positive results, manufactures findings that aren't real. Understanding how is the best inoculation against doing it yourself.

Why findings evaporate

Several forces push in the same direction, toward a literature fuller of "discoveries" than the truth can support:

• p-hacking, trying analyses until one crosses p < .05: dropping "outliers," adding a covariate, testing a subgroup, collecting a few more participants and re-checking. Each step feels reasonable in isolation. These are the questionable research practices the Ethics course dissects in detail.
• The garden of forking paths. Even without conscious fishing, the many defensible choices a researcher would have made for whatever the data happened to show inflate the false-positive rate. You don't have to run every path; it's enough that the path you chose depended on the data.
• Publication bias. Journals prefer significant, novel results, so the file drawer fills with null findings that never see daylight. The published record is a biased sample of what was actually found, the same asymmetry a meta-analysis funnel plot is designed to expose.
• Low power. Underpowered studies not only miss real effects, they make the significant results they do produce more likely to be flukes and to badly overestimate the true effect size, an inflation that follows directly from the effect size and power arithmetic.
• HARKing — Hypothesizing After the Results are Known: presenting a pattern you found by exploration as though you'd predicted it all along, which turns a chance blip into a "confirmed" theory.

Play the garden of forking paths

Below is a study with no true effect — two groups drawn from the same population, so honestly there is nothing to find. But you have four ordinary analytic choices to make (which outcome measure, whether to trim outliers, whether to adjust for age, whether to focus on the "engaged" subgroup). That's 2⁴ = 16 analysis paths. Go hunting for p < .05, then read the headline showing how often pure noise hands you a "significant" result once you're free to pick the path.

The lesson lives in the gap between two numbers. Commit to one analysis before seeing the data and your false-positive rate is exactly what statistics promises — 5%. Reserve the right to choose among these four small forks after the fact, and it climbs to roughly one in four. Add more forks (a fifth outcome, a second cut-off, an interaction to probe) and it keeps rising. No single choice was dishonest; the flexibility itself was the problem.

The crisis is science working

It's tempting to read all this as "science is broken." The opposite is truer. The replication crisis was discovered, quantified, and acted on by scientists policing their own field, the self-correction that distinguishes science from dogma. The response has been concrete and constructive: preregistration to nail down the analysis in advance, bigger and better-powered samples, open data and materials, registered reports that accept a study on its design before the results exist, and a new norm of valuing a solid null over a flashy fluke. The fixes are the subject of the next lesson.

Problem 40 of the practice problems puts four of these practices into one short manuscript excerpt in which nothing is fabricated and every number is real, and asks you to find them.