The Replication Crisis
In 2015, a team of 270 researchers tried to repeat 100 published psychology experiments. Fewer than four in ten reproduced the original result. The cause was not fraud. Almost every original study was run by honest scientists using accepted methods. The cause was ordinary, well-meaning flexibility in data analysis, combined with a publishing system that rewards surprising positive results. Together they produce findings that aren't real. Knowing how this happens is the best protection against doing it yourself.
Why findings evaporate
Several forces push the literature toward more "discoveries" than the truth can support:
- p-hacking, trying analyses until one crosses p < .05: dropping "outliers," adding a covariate, testing a subgroup, collecting a few more participants and re-checking. Each step feels reasonable in isolation. These are the questionable research practices the Ethics course covers in detail.
- The garden of forking paths. Even without conscious fishing, the many defensible choices a researcher would have made for whatever the data happened to show inflate the false-positive rate. You don't have to run every path. It is enough that the path you chose depended on the data.
- Publication bias. Journals prefer significant, novel results, so null findings stay unpublished in the file drawer. The published record is a biased sample of what was actually found, the same asymmetry a meta-analysis funnel plot is designed to expose.
- Low power. Underpowered studies miss real effects. They also make the significant results they do produce more likely to be flukes and to badly overestimate the true effect size. The inflation follows directly from the effect size and power arithmetic.
- HARKing — Hypothesizing After the Results are Known: presenting a pattern you found by exploration as though you'd predicted it all along, which turns a chance blip into a "confirmed" theory.
Play the garden of forking paths
Below is a study with no true effect: two groups drawn from the same population, so there is nothing to find. But you have four ordinary analytic choices to make (which outcome measure, whether to trim outliers, whether to adjust for age, whether to focus on the "engaged" subgroup). That's 2⁴ = 16 analysis paths. Look for p < .05, then read the headline, which shows how often pure noise gives a "significant" result once you're free to pick the path.
🌱 The Garden of Forking Paths
Same null dataset, sixteen defensible analyses. Flip the choices, hunt for significance, and watch the false-positive rate climb.
All 16 paths for this dataset. Green marks a path that reached p < .05; click one to jump there.
Compare two numbers. Commit to one analysis before seeing the data and your false-positive rate is 5%, the significance level you chose. Choose among these four small forks after the fact, and it climbs to roughly one in four. Add more forks (a fifth outcome, a second cut-off, an interaction to probe) and it keeps rising. No single choice was dishonest. The flexibility itself was the problem.
The crisis is science working
It's tempting to read all this as "science is broken." But scientists themselves discovered the replication crisis, measured it, and acted on it, and that kind of self-correction distinguishes science from dogma. The response has been concrete. It includes preregistration to fix the analysis in advance, bigger and better-powered samples, open data and materials, and registered reports that accept a study on its design before the results exist. There is also a new norm of valuing a solid null over a flashy fluke. The next lesson covers these fixes.
Researcher degrees of freedom. The umbrella term for every small, defensible choice you make while analyzing data: exclusions, transformations, which covariates, which of several measures, when to stop collecting. Each is reasonable on its own, but together they create the forking paths. The fix is to make these choices, and declare them, before you see which one gives the answer you want.
Why it matters: the person your methods most need to protect against is usually not a fraudster. It is you: hopeful, pattern-seeking, and six months into a project you badly want to succeed. So the safeguards have to be built into the design: preregister, power your study, share your data, and treat an exploratory finding as a hypothesis to test next time, not a result to announce now.
Problem 97 of the practice problems hides four of these practices in a single paragraph of a write-up that invents nothing, then asks you to find them.
Common questions
Is p-hacking always intentional fraud?
Usually not. Most p-hacking is motivated flexibility rather than deliberate deceit. An honest researcher who wants a project to work makes a string of individually defensible choices (dropping an 'outlier', adding a covariate, checking a subgroup) that happen to push p below .05. This is the 'garden of forking paths': the analysis you'd have run depended on the data, so the true false-positive rate is far above 5%. It's a systemic problem to design against with preregistration, not a character flaw to accuse people of.
Does a failed replication mean the original finding was wrong?
Not on its own. Treating one failed replication as a verdict repeats the mistake that produced the problem: reading a single study as definitive. Several things can produce a non-replication. The original may have been a false positive, the possibility most people assume first. The replication may itself be underpowered, in which case it has simply failed to detect something. The effect may be real but smaller than first reported, so a replication powered for the inflated original estimate misses it. Or the effect may depend on something that differed between the two studies, such as the population, the procedure or the time. That would be a finding to follow up, not a failure. The question is settled by accumulated evidence: several well-powered replications, ideally preregistered, pooled in a meta-analysis, not by an argument about whose study was better.
How many psychology studies actually replicate?
In the landmark 2015 Reproducibility Project, the Open Science Collaboration repeated 100 published psychology studies. Only about 36–39% produced a significant result in the same direction, and the replication effect sizes were roughly half the originals'. Rates vary by field and by how 'replication' is scored, and later large-scale projects found similar or somewhat better figures. The exact number matters less than the pattern: a sizeable fraction of published findings don't hold up, and the fix is better methods, not more accusations.